Atria, a research agent that tries to prove itself wrong

I gave an AI research agent called Atria one job. Test a thesis, try hard to break it, and write up what survives as a preprint, with no money spent. It ran on free Kaggle GPUs and a regular CPU.

The first paper, The Cost of Relocation, tested a claim that training without gradients costs more in direct proportion to model size. The cost did grow, from 831 times the cost of normal training at the smallest size to about 11,300 times at the largest, but more slowly than the claim predicted. A second claim, that low-rank changes help, was refuted. The models were tiny, 4,096 parameters at most, and one claim in the paper was later retracted.

The second paper, The Plateau Is Not the Floor, asked why forward-only fine-tuning stalls. The answer was that the search runs out of queries. The stall level moved by 13 orders of magnitude when the number of search directions changed 32 times, and the paper openly corrects the first one.

The third piece is a report. It shows that the effective dimension of this kind of fine-tuning can be measured as training runs, and the formula held within 6 percent after the first 10 steps. The decisive test on a real 70-million-parameter language model is planned and scripted, and has not run yet.