Standard guidelines for building large language models (LLM) optimize only for training costs and ignore inference costs. This poses a challenge for real-world applications that use inference time scaling techniques to improve the accuracy of model responses, such as drawing multiple reasoning samples from a model during deployment.
To close this gap, researchers from the University of Wisconsin-Madison and Stanford University presented From Training to Testing (T2) scaling laws, a framework that jointly optimizes the parameter size of a model, the size of its training data, and the number of output samples during testing.
In practice, their approach proves that it is computationally optimal to train significantly smaller models on more data than traditional rules require, and then use the saved computational overhead to generate many resamplings in savings.
For enterprise AI application developers training their own models, this research provides a proven blueprint for maximizing return on investment. This shows that AI reasoning does not necessarily require a huge expenditure on boundary models. Instead, smaller models can perform more strongly on complex tasks, while the per-query inference cost can be managed within real-world deployment budgets.
Conflicting scaling laws
Scaling laws are an important part of developing large language models. Pre-designing scaling laws dictates the best way to allocate computations during model creation. test time scaling laws guide how computation is distributed during deployment, such as allowing the model to “think longer” or generating multiple reasoning patterns to solve complex problems.
The problem is that although these scaling laws are fundamentally intertwined, they were developed quite independently of each other.
A model’s parameter size and training time directly dictate both the quality of its inference samples and its cost per query. Currently, it is the industry gold standard for preparation Chinchilla rulethis suggests a computationally-optimal ratio of about 20 training tokens per model parameter.
However, the creators of modern AI model families such as Llama, Gemma, and Gwen regularly break this rule, deliberately overloading their small models with large amounts of data.
As Nicholas Roberts, co-author of the paper, told VentureBeat, the traditional approach breaks down when building complex agent workflows: "In my opinion, the result stack is broken when each individual inference call is expensive. This is the case when the models are large and many resampling needs to be done." Instead of relying on massive models, developers can use overengineered compact models to perform this resampling at a fraction of the cost.
But since the scaling laws of training and testing are examined in isolation, there is no rigorous framework for calculating how much a model should be overtrained based on how many reasoning examples it should generate during deployment.
As a result, no formula has previously existed that jointly optimizes model size, training data size, and testing time savings budgets.
The reason this framework is difficult to formulate is that the pre-test and test time scales speak two different mathematical languages. A model’s performance during pre-training is measured using “loss”, a smooth, continuous metric that tracks prediction errors as the model learns.
During testing, developers use real-world, downstream metrics to evaluate a model’s reasoning capabilities, such as pass@k, which measures the probability that the model will produce at least one correct answer among k independent, repeated attempts.
Experiment with scaling laws
To address the relationship between training and deployment, researchers introduce Training to Test (T).2) scaling laws. At a high level, this framework predicts the inference performance of a model by treating three variables as a single equation: the size of the model (N), the amount of training tokens it learns (D), and the number of reasoning patterns it generates when inferring (k).
T2 combines the pretraining and inference budgets into an optimization formula that takes into account both the base cost (6ND) to train the model and the complexity cost to re-query while inferring. Researchers have tried different modeling approaches: modeling pre-training loss or performance at test (pass@k) as functions of N, D, and k.
The first approach takes the familiar mathematical equation used for Chinchilla scaling (which calculates the model’s prediction error or loss) and directly modifies it by adding a new variable that calculates the number of retest time samples (k). This allows developers to see how increased output computation lowers the model’s overall error rate.
The second approach directly models the downstream pass@k accuracy. It tells developers the probability that their application will solve the problem given a given computing budget.
But should enterprises use this framework for every application? Roberts clarifies that this approach is highly specialized. "I imagine you won’t see much benefit for knowledge-intensive applications like chat models." he said. Instead, "T2 adapted to applications that require reasoning, such as coding, where you would normally use resampling methods as your test time measurement method."
What it means for developers
To confirm T2 to expand the scale of the laws, the researchers built a large test bed of more than 100 language models ranging from 5 million to 901 million parameters. They trained 21 new, highly trained checkpoints from scratch to test whether their mathematical predictions held true. They then compared the models on eight different tasks that included real-world datasets such as SciQ and OpenBookQA, along with synthetic tasks designed to test arithmetic, spatial reasoning, and knowledge retention.
Both of their mathematical models proved that the computationally optimal boundary deviates sharply from the standard Chinchilla scale. To maximize performance under a fixed budget, the optimal choice is a model trained on more data that is significantly smaller than the traditional 20-token-parameter rule dictates.
In their experiment, the small overtrained models consistently outperformed the larger, Chinchilla-optimal models on all eight estimation tasks when sampling costs were considered during testing.
The technical barrier for developers who want to apply these findings is surprisingly low.
"With our existing models, nothing is required to measure the trial period," Roberts said. "During deployment, developers can fully integrate infrastructure that makes the sampling process more efficient (for example, KV caching if you use a transformer)."
KV caching helps maintain previously processed context so that the model does not have to re-read the original query from scratch for each new reasoning instance.
However, overtraining comes with practical trade-offs. Although overtrained models can be stubborn and harder to fine-tune, Roberts notes that when they apply controlled fine-tuning, "although this effect was present, it was not a strong enough effect to drive the optimal model back to Chinchilla." The optimal computational strategy remains strongly biased toward compact models.
However, teams pushing this to the absolute limit should be wary of hitting their physical data limits. "Another aspect is that if you take our exercise recommendations to extremes, you may actually run out of exercise information." Roberts said, referring to those approaching "information wall" where high-quality internet data is exhausted.
These experiments confirm that aggressively overtraining a compact model is the most efficient way to spend a computational budget that is practically and mathematically finite if the program relies on generating reasoning patterns over multiple tests.
To help developers get started, the research team plans to open their own checkpoints and code soon, allowing enterprises to input their data and test scaling behavior immediately. Ultimately, this framework acts as an equalizing force in the AI industry.
This is especially important because the high cost of boundary models can be a barrier when you scale agent applications based on reasoning models.
"T2 fundamentally changes who can build strong reasoning models," Roberts concludes. "You may not need huge computing budgets to get state-of-the-art reasoning. Instead, you need good data and a smart allocation of your training and inference budget."
