The decay phase that was missing

The most consequential change is also the least conspicuous, and it began as a defect rather than an idea.

Argonne 3.0 used a warmup–stable–decay learning-rate schedule. The learning rate governs how large a correction the model makes at each of its hundreds of thousands of update steps; the standard practice is to raise it quickly, hold it at a plateau for the bulk of training, and then anneal it toward zero at the end, so that the final weights settle rather than continue to jump around. The decay phase is where a large part of the quality gain lives.

Argonne 3.0 never performed it. Its launcher passed a decay length of zero, and the scheduler interpreted zero as “return a constant” — so the plateau ran to the last step of the run and the model was exported at full learning rate. The setting intended to soften the ending silently removed it.

Figure 2: Argonne 3.0’s schedule is reconstructed from its launcher flags and the scheduler source; Argonne 3.5’s is read from its own training logs. Vertical dashed lines mark 3.5’s stage boundaries.

The correction is worth its own measurement, because it was made mid-flight: Argonne 3.5’s first stage did decay, its second stage inherited the same zero and did not, and it was corrected while that stage was running. Held-out cross-entropy was measured on eight subject-matter categories before and after the first stage’s decay phase: every one improved by 0.12 to 0.34 nats, including categories the stage never trained on, and that decay phase alone out-earned the entire 175,000-step plateau preceding it. That is the size of the effect Argonne 3.0 forfeited.

The second stage made the case concrete in the other direction. Measured mid-stage, while still at constant learning rate, it was improving on every one of its target subjects and simultaneously losing general knowledge: held-out cross-entropy on general educational text rose by 0.25 nats, far enough that Argonne 3.0-base was briefly the better model on it, despite that text making up 9% of the mixture — the other 91% simply swamped the replay. Restoring the decay reversed it, and by the first post-decay checkpoint all eight categories were ahead of Argonne 3.0-base again.

Two further observations belong with this. During a constant-rate plateau the weights are demonstrably not settling: five checkpoints taken 1,200 steps apart swing 0.62 nats on the same held-out text, simply tracking whatever the most recent batches contained. And the identical defect later appeared in a successor project, because the fix had been applied in one working copy of the launcher and not the other — a reminder that a repository with two checkouts has two configurations.

Numeracy designed in, rather than repaired afterwards

Argonne 3.0-base was trained on FineWeb, a filtered snapshot of the open web. It emerged fluent and broadly knowledgeable and unable to compute 8 + 3. The reasoning work that followed spent months repairing this: a corrective training stage on mathematical text, then a linear average of two checkpoints to reconcile the mathematics it gained against the general knowledge it lost.

Argonne 3.5 mixed mathematics into pretraining from the first step instead — FineMath at 15% of the corpus, FineWeb at 85% — and then followed it with two further stages on progressively more specialised data.

Figure 3: Bar length is tokens trained in that stage; the label gives the token count and the context length at which the stage ran. Argonne 3.5 trained on 17% more text, in three stages rather than two.

The second stage, an anneal, trains on a mixture of code, mathematics, reasoning traces and tool-use transcripts, with a smaller replay tier of general educational text to guard against forgetting. The third stage takes a deliberately disjoint quarter of that same corpus — reserved before the second stage began, so no reasoning token is trained on twice — and reads it at a context of 13,568 tokens instead of 1,024.

A cheap probe tracked whether the mathematics was arriving: 20 arithmetic and word-problem questions and 15 general-knowledge questions, graded by exact match, run on intermediate checkpoints for about ten GPU-minutes each. Its purpose was a go/no-go decision — whether the base was worth the expense of a reasoning recipe — not a capability measurement.

Figure 4: A 35-item probe, greedy decoding, run on live checkpoints. The probe has a measured noise floor of about two items and saturates, so it certifies readiness rather than measuring capability.

A controlled comparison isolates the cause. The same probe, the same grader, the same architecture and tokenizer were applied to a freshly downloaded Argonne 3.0-base: 3 of 20 on mathematics, 14 of 15 on general knowledge. At the point Argonne 3.5 crossed the threshold it had seen roughly 5.8 billion interleaved FineMath tokens, and that interleaving was the only substantive difference between the two corpora. The remaining misses are also different in kind: Argonne 3.0-base failed single-digit arithmetic, while Argonne 3.5’s failures are multi-step and formulaic — solving for a variable, a rectangle’s perimeter, a factorial.

Two honest limits. The probe is small, curated and saturating, and a separate few-shot evaluation on standard benchmarks read close to chance on this checkpoint, so it should not be read as a claim of broad capability. And no standard held-out benchmark suite has yet been run on the released base.

The training record

Figure 5: Median loss within each short window of the run; the band spans the 10th to 90th percentile inside the window, which is a measure of how much consecutive batches differ from one another.

The width of the band is itself a finding. A corpus that interleaves web text and mathematics produces batches of visibly different difficulty, so per-step loss is noisy by construction — anything from 1.7 to 3.2 was normal in stage 1, with the mathematics shards at the bottom of the range. No divergence occurred, and no non-finite loss was recorded at any point in the run.

Context that is trained, not extrapolated

Argonne 3.0 used rotary position embeddings with a large wavelength parameter (θ = 10⁶), which is often assumed to let a model generalise beyond the sequence length it was trained on. On this architecture it does not. Measured directly, by comparing the checkpoint that entered the context-extension stage against the weights it produced:

Figure 6: Held-out arXiv text, negative log-likelihood by token position; lower is better. The vertical dashed line marks the length the final stage actually trained on.

Three properties of this result matter. Before the final stage the model is coherent within its 1,024-token window and almost useless beyond it, with the break landing exactly at the pretraining length — extrapolation did not happen at all. After the stage, prediction improves monotonically with position and keeps improving past 13,568 tokens, which is the signature of a model that has learned to use distance rather than one that has merely been shown longer inputs. And the short-context control improved, so nothing was traded away; at an earlier point in the same stage that control had been slightly worse, and the terminal decay recovered it.

Eight-bit matrix multiplication

The one throughput change was to compute the large matrix multiplications in 8-bit floating point while keeping the master weights in 32-bit. A dedicated search measured this before it was adopted: an individual matrix multiplication runs about 1.83× faster, but only about 44% of a training step is spent in eligible operations, so the end-to-end gain is about 1.25× — at indistinguishable quality and with no non-finite values in any run. Two conditions were not optional. The scaling arithmetic must be fused into the multiplication by the compiler, since running it interpreted is roughly 2.3× slower than not using FP8 at all; and the vocabulary had to be padded from 151,669 to 151,680 so that the output projection was eligible too, which the export then trims back.

Two other candidate speedups were tested and rejected. Recomputing the loss in chunks to free memory made the run 25% slower at a context of 1,024, where the step is compute-bound rather than memory-bound. And the attention implementation, which the startup banner reported as a slow fallback, was measured at the real tensor shapes and found already to be running the fast kernel — the banner was a static string, not a runtime observation. Both investigations concluded that the configuration was already near-optimal, which is a useful thing to be able to establish.

The same audit turned up a finding that belongs here rather than in a footnote. The architecture includes an interleaved local-attention pattern, in which alternate layers attend only to a 256-token neighbourhood — one of the features the earlier article credited to Argonne 3.0. It has never actually run. The kernel that implements windowed attention is absent from the installed software, so the model silently falls back to full attention and the window is ignored, in every production run of both versions. Nothing was lost, because the recipe search measured local attention as quality-neutral at a context of 1,024 — but a documented architectural feature was inert for two model generations, and only a direct measurement at the tensor level revealed it.

What it actually takes to run for a month

The run occupied 108 scheduler allocations of two or three GPUs each, submitted as a self-resubmitting chain that saves a checkpoint and exits cleanly before each wall-clock limit.

Figure 7: Each horizontal segment is one scheduler allocation. Horizontal gaps are queueing, deliberate pauses, and two chain failures. Colour marks how the allocation ended.

Half the elapsed time was not computation. Some of that is queueing on a shared machine and some was deliberate — the GPUs were lent to other work — but two stretches were failures of the chain itself, and both are worth recording because neither produced an error message.

The first is visible above. A slice was pre-empted by the scheduler, which delivers an external termination signal; the resubmission logic fires on clean exit, on its own wall-clock warning, and on a non-zero crash, and an external cancellation is none of those. The chain simply stopped, having lost no progress, with nothing queued and nothing logged. The second was a nightly placeholder job whose name collided with an unrelated interactive job of the same name, so it was cancelled by its own owner nineteen seconds after starting.

Two sizing lessons came out of the same run, both instances of measuring the wrong moment. Host memory was set from the observed steady state of 115 GiB, but the true peak is the resume transient, when all three processes load a 34 GB checkpoint simultaneously — about 146 GiB, which had been passing with four gigabytes to spare. Separately, a checkpoint-extraction job sized from a reported peak of 23.4 GB was killed at 27.0 GiB, because periodic sampling misses a short allocation spike. The reliable rule is to size such jobs from the checkpoint file, not from an observation.

Ceiling and floor

Isolating the contribution of each stage requires holding everything else fixed, which the following sequence does: first the base is swapped under an unchanged recipe, then the chain-of-thought data is swapped on the new base, then the final weight average is applied.

Three measurements are reported at each step, and the distinction between them is the whole point. Greedy accuracy is what a user gets from one attempt — the deployable number. Self-consistency samples eight answers and takes the majority. pass@8 counts a problem as solved if any of the eight attempts is right; it measures what the model is capable of, not what it reliably does.

Figure 8: Clean SVAMP word problems, n = 300, K = 8, one grader. Each step changes exactly one thing relative to the step before it.

The first step is the counter-intuitive one and it replicated three separate times: a better base raised what the model could produce by roughly fifteen points and what it did produce by essentially nothing. Better pretraining bought latent capability that a single greedy attempt could not reach.

The reason it could not reach it was not that the answers were wrong. They were absent.

Figure 9: Share of greedy attempts that produced no answer at all, before and after the chain-of-thought data was restricted to short, closed, correct traces. n = 300 per problem set.

This is the central mechanism of the reasoning line, and it is a training-data property rather than a decoding one. The model was writing reasoning traces that never terminated, so no answer was ever emitted. Training exclusively on traces that are short, properly closed, and correct makes termination a property of the weights: the failure rate collapses by a factor of thirty to forty, and the decode-time patch that had previously been the only effective remedy becomes worthless because there is nothing left for it to fix. The run that produced this took eighteen minutes on three GPUs.

Two details of that stage were determined by ablation rather than assumption. A second epoch of the same data is worse — no individual difference is significant, but greedy accuracy falls on both problem sets and non-termination roughly doubles, and that consistency is the signal. And roughly 30% of the mixture consists of direct answers with no reasoning at all; a configuration flag that silently discards those rows is what caused the general-ability regression in the earlier line.

Preference tuning, by contrast, was measurably inert here: its loss sat at ln 2 — the value corresponding to no preference learned at all — with a reward margin of about 0.001. The final weight average is therefore effectively a blend with the instruction-tuned checkpoint, and it earns its place: about two points of mathematics, and the repair of the one general-ability failure in the probe, which was a grammar-correction item. That is instruction-following arriving through 207,865 conversations by way of the average, and it is exactly the kind of thing a fact-recall probe cannot see.

The blending weight is not a free parameter. At α = 0.70 the reasoning trace stops closing again and non-termination returns, reproducing on a new base a threshold first found on the old one.

Against Argonne 3.0-think

Both models were evaluated in a single job, with the same grader, the same problems and the same seed, rather than compared against previously recorded numbers.

Figure 10: Clean SVAMP and ASDiv, n = 300, K = 8, both models scored in one job. Neither problem set appears in any training stage of either model.

The gap on a single attempt is the one that matters for use, and it is between two- and threefold. It is worth being precise about where it came from: not from reinforcement learning, not from distilling a stronger teacher, and not from a larger model. It came from a base that could do arithmetic and from 28,428 training examples chosen for the property of ending.

Traps, in the order they were encountered

Each of the following cost measurable time or compute, and each is silent — none produced an error, and several produced entirely plausible logs.

1. A schedule parameter of zero meant “no schedule”. Argonne 3.0 trained its whole 76 billion tokens at peak learning rate because a decay length of zero was interpreted as a constant. The same defect then recurred in a successor project because the fix was applied to one of two checkouts of the same launcher.
2. Two command-line defaults corrupted a third of the reasoning data, with no effect on row count, step count or loss.
3. Benchmarks can be uniformly blind to a severe regression. Five multi-step problem sets, three seeds and a general-ability probe all passed while single-step arithmetic was failing half the time.
4. One evaluation set was unusable and another leaked. GSM8K is contaminated for every model on this line and is never reported; MATH-500 carries measured near-duplicate leakage, which is invisible to exact-match decontamination and had to be quantified.
5. pass@8 cannot separate two models. Re-running the identical model, seed and settings reproduced greedy accuracy and self-consistency exactly and moved pass@8 by 5.7 points.
6. One seed decides nothing. Two runs of the identical fine-tuning recipe differ by 1.7 points on a five-set mean and by up to 12 on a small probe. Two conclusions on this line were withdrawn for having been read from one seed.
7. A throttled GPU is invisible in the log. One node ran at 385 MHz of a possible 1,785 and 163 W of a possible 400, making a job 3.6× slower with no error. The clock and throttle reasons should be checked before the configuration is blamed.
8. The effective batch size determines the number of GPUs, not the reverse. An effective batch of 20 does not divide across three processes, and an unnoticed change silently breaks the pairing between batch size and learning rate.
9. A configuration that is correct locally can be wrong when published. The release build initially omitted the field that lets the model be loaded from the Hub at all, capped the trained 13,568-token context at 4,096, and set an end-of-sequence token that would have prevented generation from ever stopping. All three were correct for the local harness — which is exactly why no amount of evaluation could have found them. Only a diff against the artefact users receive did.
10. The generated model card described a different model, with the wrong layer count, the wrong hidden size, the wrong base model and no mention of reasoning. A release artefact needs reading before publication exactly like the weights do.
11. A documented architectural feature never ran. The interleaved local-attention window is silently ignored, because the kernel implementing it is not installed. A startup banner reporting the attention path was a fixed string and could not have revealed it.
12. An automatic checkpoint-pruning rule deleted 14 of 17 checkpoints overnight. Retention policy belongs to the owner of the run.

Where the compute went

Figure 14: Measured from the scheduler accounting database. The horizontal axis is logarithmic; the three non-base runs together are 1.8% of the total.

The asymmetry is the practical argument of the whole project. The artefact people interact with is produced by three fine-tuning runs that together take less than a day on a few GPUs; what determines how good it can be was decided a month earlier, by what went into the base.

Lessons

1. Hold the architecture fixed to learn something. Argonne 3.5 changed no component of Argonne 3.0, which is the only reason the recipe and data effects are attributable at all. The temptation in this kind of work is to redesign the model at the same time; resisting it turned a new version into an experiment.
2. Design the capability into pretraining instead of repairing it afterwards. Mixing mathematics into the corpus from the first step accomplished, at no measured cost to general knowledge, what a corrective stage plus a reconciling weight average had accomplished imperfectly before — and it removed both stages from the pipeline.
3. A default of zero is not a safe default. The single largest recipe deficiency and the single largest data defect were both parameters left unset, and both were invisible in every artefact a training run normally produces.
4. Better pretraining raises the ceiling; only post-training raises the floor. Improving the base moved what the model could produce by fifteen points and what it reliably produced by nothing. Expecting more pretraining to fix a deployment failure wasted several rounds of experiment before the distinction was measured.
5. Termination is a capability. More than half of all attempts were failing by never finishing. The fix was not a decoding trick but a training set restricted to examples that end, and every later attempt to relax that restriction lost.
6. Evaluate the thing that is missing, not the thing that is easy to measure. A suite of five respected benchmarks, replicated across seeds, was completely blind to a model that could not compute 2 + 2.
7. Operational engineering decides whether a month-long run finishes. Half the elapsed time of this run was not computation, and the two chain failures, the memory mis-sizing and the throttled GPU were each silent. None of it is interesting, and all of it is load-bearing.

Availability

• Models: argonne-3.5-base and Argonne-3.5-think on Hugging Face, alongside their Argonne 3.0 counterparts and the rest of the family.
• Code: github.com/PursuitOfDataScience/ArgonneAI — the complete pipeline, one branch per version, together with the full chronological training log from which every number above is drawn.
• Companion articles: Pretraining a Language Model From Scratch: Argonne 1.0 to 3.0 and How I Taught a Small Language Model to Reason.

The earlier article in this series ended by claiming that capability is set during pretraining and that fine-tuning only calibrates it. Argonne 3.5 was built to test that claim and largely confirms it, with one qualification that took a month to see: the base sets the ceiling, but a model can sit far below its own ceiling for reasons that have nothing to do with capability — in this case, because it never brought to a close the reasoning it had correctly begun.