How Frontier Labs Train Large Language Models

Table of Contents

• Why read the tech reports?
• The shape of a frontier model
  • The settled core
  • The big shift: dense to MoE
  • Where labs still disagree
• Data: the real moat
• Pre-training: scaling, precision, stability
• Post-training I: SFT, cold-start, and distillation
• Post-training II: RL, the engine of reasoning
• Agentic RL: credit assignment over long horizons
• Catastrophic forgetting and continual learning
• Alignment: helpfulness, safety, honesty
• Evaluation: measuring the climb
• Safety and red-teaming
• The convergent recipe
• Open challenges

Why read the tech reports?

For most of the deep-learning era, how a frontier model is actually trained was the industry’s best-kept secret: a few sentences in a system card, a parameter count, a benchmark table. You could read every paper and still not know how to build one. That has changed. Across 2024-2026 a remarkable thing happened: lab after lab published a genuine end-to-end technical report — not a teaser, but the data pipeline, the architecture ablations, the optimizer, the reinforcement-learning recipe, the reward design, the evaluation methodology, and the safety process. DeepSeek (V3, V3.2, R1), Qwen3, Kimi K2 and k1.5, Meta’s Llama 3, Google’s Gemma, Microsoft AI’s MAI-Thinking-1, Zhipu’s GLM-4.5, Alibaba, Moonshot, Xiaomi’s MiMo, Tencent’s Hunyuan, MiniMax, NVIDIA’s Nemotron, and the fully-open OLMo 2 / Tulu 3 — and, through mid-2026, a fresh wave: DeepSeek-V4, GLM-5 / 5.2, Kimi K3, Qwen3.5, xAI’s Grok 4.5, and Thinking Machines’ Inkling. Together they are an accidental textbook.

Read them side by side and the striking thing is not how different the labs are, but how convergent they have become. Strip away the branding and almost every report walks the same path:

Thesis. By 2026 there is essentially one recipe for training a frontier LLM — a standard pipeline from data → pre-training → mid-training → post-training (SFT then RL) → alignment → evaluation → safety. What separates the labs is no longer the skeleton; it is a small set of design choices (how to balance a mixture-of-experts, which RL algorithm variant, which rewards to trust, whether to use synthetic data, whether to distill) and a handful of hard-won tricks for staying stable at scale.

This post is that recipe, taught one stage at a time. We use Microsoft AI’s MAI-Thinking-1 report as a spine, because it is unusually candid and frames the whole enterprise nicely — as building a “hill-climbing machine”: the integrated system of data pipelines, training infrastructure, RL environments, evaluation suites, and safety tests that turns model development into an empirical optimization loop. At each stage we ask the same question — how does MAI do it, and how does everyone else? — and let the other reports agree, disagree, and occasionally contradict each other.

Figure 1. The recurring pipeline this post follows. Data is curated, the base model is pre-trained and then mid-trained, post-training adds SFT and several RL “climbs” that are consolidated into one model, and everything is measured by evaluation and red-teaming — whose results feed the next iteration. Every lab in this post instantiates this same skeleton; they differ mainly in the knobs.

A note on how to read these reports skeptically. A technical report plays two roles at once. It is a product announcement — so the benchmark tables are chosen to flatter — and it is a reproducible recipe — so the methods sections are where the real signal lives. The fully-open efforts (OLMo 2, Tulu 3, Nemotron) disclose what the others must paraphrase, so we lean on them whenever a closed report gets vague. And throughout, keep one distinction in mind that we will return to constantly: the difference between what a lab claims helps and what it actually ablated and measured. The good reports are mostly the latter.

Takeaway. The 2024-2026 tech reports have converged on a single end-to-end recipe; this post teaches that recipe stage by stage, using MAI-Thinking-1 as a spine and the other labs as a cross-checking chorus.

The shape of a frontier model

Before the pipeline, the artifact. If you opened the config of MAI-Thinking-1, DeepSeek-V3, Qwen3, Kimi K2, Llama 3, and Gemma 3 side by side, you would be struck by how similar they are. The decoder-only Transformer has converged to a near-universal block, and the 2024–2026 reports treat that block as boilerplate — they spend their architecture sections on the two things that are still live: how to be sparse (mixture-of-experts) and how to make attention cheap at long context.

The settled core

Every model in this post is a decoder-only Transformer built from the same five parts, each of which “won” a years-long bake-off:

• RoPE for position — rotary embeddings encode relative distance and extrapolate cleanly, which is why all of them then stretch context with RoPE-base scaling / YaRN (Su et al., 2021).
• GQA for the KV-cache bottleneck — grouped-query attention gets most of full attention’s quality at a fraction of the decode-time memory (Ainslie et al., 2023).
• SwiGLU for the feed-forward layer — a gated activation that buys free quality at fixed FLOPs (Shazeer, 2020).
• RMSNorm for normalization — LayerNorm quality without the mean-centering cost (Zhang & Sennrich, 2019) — now routinely paired with QK-norm (RMSNorm on the queries and keys) and a small z-loss, the two cheap stabilizers that a small-scale study showed prevent the attention-logit and output-logit blowups that kill big runs (Wortsman et al., 2023).

Consensus. RoPE + GQA + SwiGLU + RMSNorm + QK-norm is the modern decoder block. MAI-Base-1 is a textbook instance of it; so is essentially every other base model here. The only block-level knobs left are norm placement (pre-norm vs Gemma’s and OLMo 2’s pre+post / reordered norm) and how aggressively attention is sparsified.

The big shift: dense to mixture-of-experts

The real architectural story of this era is the migration from dense models to mixture-of-experts (MoE): replace the FFN with many “expert” FFNs and route each token to a few of them, so total parameters (capacity, which holds knowledge) decouple from active parameters (the per-token compute). The design that everyone copied came from DeepSeekMoE (Dai et al., 2024): two ideas, fine-grained expert segmentation (slice the FFN into many small experts and activate more of them — combinatorially more routing options at fixed FLOPs) and shared-expert isolation (a always-on expert that absorbs common knowledge so the routed experts can specialize). DeepSeek’s ablation is quotable: disabling the shared expert spikes loss, and a fine-grained model degrades more when you remove its top experts — evidence the experts really did specialize.

By 2026 this is the default. DeepSeek-V3 runs 1 shared + 256 routed (8 active) experts; Kimi K2 pushes it to 1.04T total / 32B active across 384 experts; Qwen3 drops the shared expert; MAI-Thinking-1 interleaves high-sparsity MoE layers with dense FFN layers (and finds that pairing beats medium-sparsity-everywhere on wall-clock); Llama 3 is the conspicuous dense holdout, choosing a 405B dense model explicitly “to maximize training stability.”

Figure 2. The converged decoder block (left) and the dense→MoE feed-forward shift (right): a token is routed to a few fine-grained experts plus an always-on shared expert. Labs differ in expert count, top-k, whether a shared expert exists, and how layers interleave dense and MoE.

Where labs still disagree

Two axes remain genuinely contested, and they are where the engineering effort goes.

Attention efficiency. GQA is the baseline, but the frontier is a menagerie of ways to shrink the KV cache or the quadratic cost: DeepSeek’s Multi-head Latent Attention (MLA) (DeepSeek-V2) compresses KV into a low-rank latent (a smaller cache than GQA at better quality), later extended with DeepSeek Sparse Attention to make long-context attention sub-quadratic (DeepSeek-V3.2); Gemma 3/4 and MAI interleave 5 local : 1 global attention layers so only every sixth layer pays the long-range cost (Gemma 4 keeps this and adds Proportional RoPE, unified KV on global layers, and its first MoE variant — Gemma Team, 2026); MiniMax-M1 goes furthest with a 7:1 lightning(linear)-attention hybrid that makes 1M-token context — and cheap long-CoT RL — affordable; Hunyuan combines GQA with cross-layer attention for ~95% KV savings; gpt-oss adds attention sinks. MAI even drops positional encoding entirely on its global layers (NoPE), finding it as good as RoPE but cheaper. By 2026 this had become the race: sparse/compressed attention plus 1M-token context is now table stakes — DeepSeek-V4 ships a Compressed-Sparse + Heavily-Compressed Attention hybrid, GLM-5 adopts DeepSeek’s DSA (and GLM-5.2 adds “IndexShare” to cut 1M-context FLOPs ~2.9×), all chasing the same goal of long context that’s cheap enough to train RL on, not just serve. By mid-2026 the boldest bet — linear/hybrid attention — went mainstream at the frontier: Kimi K3 ships Kimi Delta Attention (KDA), Qwen3.5 a Gated DeltaNet hybrid (400B-class quality at ~17B-active speed), alongside MiniMax’s lightning attention — all trading a little quality for near-linear long-context cost. Sparsity also went extreme: Kimi K3 activates just 16 of 896 experts and balances them with Quantile Balancing (allocate from router-score quantiles, no aux-loss heuristic), claiming ~2.5× K2’s scaling efficiency.

MoE load balancing. Routed experts must stay balanced or training collapses and GPUs idle. There have been three eras of the same problem: the original auxiliary-loss (add a balance penalty to the objective — GShard); DeepSeek’s auxiliary-loss-free scheme (move balancing out of the gradient into a per-expert routing bias, which gives better quality and more specialization, Wang et al., 2024); and Qwen’s global-batch aggregation insight that the bug nobody noticed was computing the balance loss per micro-batch, which silently destroys expert specialization (Qiu et al., 2025).

Divergence — what you balance over matters more than how. MAI runs a GShard-style loss but aggregates expert frequencies across the global batch, and reports the punchline directly: “the aggregation strategy matters much more than the load-balancing-loss type.” So the modern answer is less about loss-vs-bias and more about balancing over a diverse enough population of tokens.

One last trick worth knowing because it recurs: multi-token prediction (MTP), where the model is trained to predict the next few tokens. DeepSeek-V3 introduced it for denser training signal and a free ~1.8× speculative-decoding speedup; MiMo and Nemotron adopt it. The standout optimizer story — Muon and Kimi’s MuonClip — belongs to the next section.

Table 1. The architecture has ~80% converged; the remaining knobs are MoE shape and attention efficiency, and that is exactly where each lab spends its cleverness.

Takeaway. The block is settled (RoPE/GQA/SwiGLU/RMSNorm/QK-norm); the live architecture game is mixture-of-experts shape and cheap long-context attention, where a handful of distinct bets (MLA, local/global, linear/lightning, sparse) coexist.

Data: the real moat

If architecture is ~80% commoditized, data is where models actually differ — and, not coincidentally, it is the stage every lab is most guarded about. The closed reports give you a token count and a sentence (“a diverse mixture of public and licensed data”); the fully-open recipes (OLMo 2, Tulu 3, Nemotron) give you the whole funnel. Put together, the pipeline is remarkably consistent: a raw crawl is filtered and deduplicated down by more than an order of magnitude, then re-mixed by a quantitative recipe.

Figure 3. The data funnel, with MAI-Thinking-1’s disclosed numbers as the concrete example: ~1.2T crawled pages are filtered for policy and quality, deduplicated several ways, and re-mixed into the ~30T training tokens. Every lab runs some version of this; the percentages and the synthetic-data policy are where they part ways.

Extraction is underrated. The content labs most want — math, code, tables — is exactly what naive HTML-to-text pipelines mangle. So the good reports describe bespoke extractors: MAI normalizes MathML and LaTeX to Markdown and uses an LLM that may only keep or remove spans (never add synthetic text); Llama 3 built a custom parser that preserves code/math structure and even keeps image alt text for equations; MiMo and Llama both note that generic filters wrongly throw away math/code-heavy pages. The payoff shows up in purpose-built corpora — StarCoder2 / The Stack v2 for code, Nemotron-CC-Math for mathematics — where careful extraction is most of the value.

Deduplication is multi-stage and load-bearing. The reports converge on a stack: boilerplate removal, exact (hash) dedup, MinHash-LSH fuzzy dedup (~0.8 similarity), templated-page skeletonization, and increasingly embedding/semantic dedup in the lineage of SemDeDup and D4, which showed you can drop ~half of web data with no quality loss, and that smart repetition beats random fresh tokens. MAI runs all of these plus a cross-dataset drop-order so the same document isn’t counted twice across sources; it reports the funnel explicitly (1.2T pages → 794B after filtering → 423B after exact dedup → ~190B after fuzzy).

Filtering and categorization turn a pile into a controllable corpus. Labs score every document with cheap classifiers — fastText and embedding models for language, topic, educational value/level, and quality — so the corpus becomes a set of labeled buckets they can mix. Essential AI’s Essential-Web takes this to its logical end: label the whole web once with a distilled taxonomy classifier, then curate any domain with a SQL-style filter instead of training a new classifier each time — exactly MAI’s “organize the corpus into interpretable dimensions” philosophy.

Data mixing has become a quantitative discipline. The domain mix (how much web vs code vs math vs multilingual) strongly determines capability, and labs no longer set it by hand. Data Mixing Laws (Ye et al., 2024) showed validation loss is a predictable function of the mixture proportions — fit it on small “swarm” runs, then optimize; RegMix and OLMix productionize this (OLMix adds reuse for evolving domain sets). MAI ran 183 models across 3 scales over 61 mixtures to map the Pareto frontier; Llama 3 picked roughly 50% general / 25% math-and-reasoning / 17% code / 8% multilingual via scaling-law experiments; MiMo ran a deliberate 3-stage mixture that ramps math+code to ~70%.

Open question — small-scale rankings can lie. The appealing assumption behind cheap mixture search is rank invariance: if mix A beats mix B at small scale, it beats it at large scale. MAI reports this breaking — a code-heavy and a STEM-heavy mixture swapped order between 5B and 23B models. Mixtures may need to be chosen by their scaling behavior, not a single small-scale bake-off.

The sharpest disagreement in this whole post lives here:

Divergence — synthetic vs human data. MAI takes the contrarian hard line: no LM-generated synthetic data in pre-training, and an active effort to detect and remove AI-generated content from the crawl (a bet that clean human data avoids a model-collapse / homogenization trap). The opposite pole is everywhere too: Hunyuan-Large trains on ~1.5T synthetic tokens from a 4-step generate-evolve-filter pipeline; Persona Hub scales synthetic diversity with a billion personas; Qwen and Nemotron lean on synthetic rephrasing and distillation. MiniMax splits the difference (avoids synthetic in pre-training, like MAI). This is genuinely unresolved, and it is the cleanest “the labs disagree” moment to flag.

A newer data frontier is worth flagging: real agent traces. xAI’s Grok 4.5 (July 2026) was mid-trained on anonymized Cursor developer-session traces — the actual keystrokes of engineers navigating and editing real codebases — to sharpen coding and agentic behavior, and Artificial Analysis ranked it #1 on agentic tool-use. It’s the clearest instance of a “compute + traces” paradigm: once you’ve scaled compute, the scarce input becomes recordings of the target behavior. (It also raises unresolved data-rights questions — whose code, under what consent.)

Finally, decontamination — keeping eval benchmarks out of training — is the quiet crisis under all of this. As benchmarks leak onto GitHub and into crawls, contamination produces flattering, fake numbers. Labs handle it coarsely: MAI removes all huggingface mirrors and applies universal 20-gram fuzzy dedup, and — the move everyone is converging on — relies on private, held-out benchmarks they are confident aren’t on the web. We return to this in Evaluation.

Takeaway. Data is the least-shared, highest-leverage stage: a converged funnel (extract → dedup → classify → mix) sits atop two unresolved questions — how much synthetic data to trust, and whether your eval set has already leaked into training.

Pre-training: scaling, precision, stability

With data in hand, pre-training is now an engineering discipline organized around three questions: how big and how long (scaling), in what number format (precision), and how to keep a months-long run from diverging (stability). Plus a fourth stage that barely existed two years ago: mid-training.

Scaling: from Chinchilla to deliberate over-training. The 2020 Kaplan laws said loss is a smooth power law in parameters, data, and compute, and recommended spending most of a budget on parameters — the era of Gopher and PaLM. Chinchilla corrected this: for a fixed training budget, scale parameters and tokens together, ~20 tokens per parameter. But 20 TPP is training-compute-optimal, not deployment-optimal — once you amortize a model over billions of inference tokens, the right move is a smaller model trained far past 20 TPP. So the field deliberately over-trains: Llama-3-8B sees ~15T tokens (~1900 TPP); MAI runs its main model at 500–1000 TPP for a compact, inference-cheap result while ablating architecture near the Chinchilla-optimal region. MAI formalizes the methodology nicely with a scaling ladder (train a family at constant tokens-per-active-parameter) and an Efficiency-Gain metric (how much more compute the baseline would need to match a candidate) so that every change is justified by its scaling curve, not a single data point.

Figure 4. Chinchilla says scale parameters and tokens together (~20 tokens/param) for the cheapest path to a loss; inference economics push labs to pick a smaller model and over-train it far to the right. Labs ablate near compute-optimal but ship deep in the over-trained regime.

Consensus — over-train for inference. Nobody trains compute-optimal anymore. The shared logic: training is paid once, inference is paid forever, so trade extra training tokens for a smaller, cheaper model. The open edge of this is the data wall — at extreme TPP you run out of unique high-quality tokens, which loops back to the synthetic-data debate.

Ablate small, ship big — and how you know it transfers. You cannot A/B-test a 1T model by training 1T models; a single run costs millions. So every architecture, data, and hyperparameter decision is made on small proxies and extrapolated — and the real skill is making that extrapolation trustworthy. Three complementary tools do the heavy lifting:

• Scaling ladders + Efficiency-Gain. Train a family of models at increasing size/compute (an IsoFLOP or constant-TPP ladder), fit the loss-vs-compute curve, and read off the target. MAI’s discipline is the template: a change is adopted only if its Efficiency-Gain persists or grows across the whole ladder — something that helps at 1B but fades by 30B is rejected as a small-scale artifact. Llama 3 pushed this furthest, fitting downstream-accuracy scaling laws to predict benchmark scores ~4 orders of magnitude out, then choosing 405B/16.5T from the fit.
• μP / μTransfer for hyperparameters. The clean answer to “how do I pick the 1T learning rate without sweeping at 1T”: re-parameterize the model (maximal-update parameterization, Yang et al., 2022) so the optimal LR and init are (near-)invariant to width — tune them on a small model and transfer zero-shot to the large one. Muon’s “match AdamW’s update RMS” trick is in the same spirit of making hyperparameters scale-stable.
• Small-scale instability proxies. You can reproduce big-run failures (loss/attention-logit spikes) at small scale by cranking the LR (Wortsman et al., 2023); that’s how QK-norm and z-loss earned their place before anyone paid for a large run.

Insight — scaling laws de-risk, they don’t guarantee. The uncomfortable truth is that transfer can break: the rank-invariance assumption (if A beats B small, A beats B big) fails often enough to be dangerous — MAI reports a code-heavy vs STEM-heavy mixture that literally swapped order between 5B and 23B — and some capabilities and instabilities only emerge at scale. So the honest interview answer to “prove it works at 1T” is: you don’t prove it, you de-risk it. Judge changes by the trend across the ladder (not one point), tune HPs with μP, verify at the largest affordable intermediate scale before committing, keep the least-transferable calls (data mixture, final LR schedule) validated as near the target as budget allows, and prefer changes with a mechanistic reason to transfer. The final jump to 1T is always partly a bet — the craft is in shrinking it.

Precision: BF16 → FP8 → FP4. Training precision has marched down from the FP16 mixed-precision era (Micikevicius et al., 2017) through BF16 (Zamirai et al., 2020) to today’s frontier. The most visible efficiency shock was DeepSeek-V3 training a 671B model in FP8 — using FP8 formats (E4M3 forward, E5M2 backward) with fine-grained tile/block-wise scaling to tame outliers — for a total of ~$5.6M, with a relative loss error under 0.25% vs BF16 (helped by tricks like stochastic rounding). NVIDIA’s Nemotron 3 pushes further to NVFP4 (4-bit), stable to 25T tokens via layer-by-layer precision rules (keep the last ~15% of the network high-precision), and gpt-oss ships MXFP4 MoE weights so a 120B model fits on one GPU. MAI trains in FP8 too. The holdouts are instructive: Llama 3 stayed in BF16 for robustness — a recurring “stability over efficiency” theme.

Divergence — the optimizer monopoly is cracking (and Muon is winning). For a decade, AdamW was the only answer. Now Muon (Liu et al., 2025) — which orthogonalizes the momentum update via a Newton–Schulz iteration and matches AdamW’s update RMS — claims ~2× compute efficiency, and the flagships are switching: GLM-4.5/GLM-5 use Muon, Kimi K2 uses MuonClip (Muon + a QK-Clip that rescales the query/key projections to cap attention logits; a 15.5T-token trillion-parameter run with zero loss spikes), and as of 2026 even DeepSeek-V4 (2026) — long an AdamW shop — adopts Muon “for faster convergence and greater training stability.” Mid-2026 Kimi K3 pushes it further with Per-Head Muon (orthogonalizing each attention head independently). AdamW still trains MAI, Qwen, and Llama, but the momentum (pun intended) is clearly with Muon — the most consequential optimizer shift in years.

Stability is its own research area. A thousand-GPU, months-long run dies from loss spikes, diverging logits, or even hardware bit-flips. The cheap, near-universal fixes — QK-norm and z-loss — come from the small-scale-proxy study (Wortsman et al., 2023), and the fully-open OLMo 2 report is the best public catalogue of the rest: a repeated-n-gram document filter to kill spike-inducing data, a std-0.02 initialization, AdamW ε lowered to 1e-8, reordered (pre+post) norm, no weight decay on embeddings — each with a measured reduction in their “spike score.” MAI’s infra layer adds determinism and silent-data-corruption handling. None of this appears in a closed system card.

Mid-training is the new stage. Between raw pre-training and post-training, labs now insert a mid-training phase that up-weights STEM/math/code on high-quality (often annealed) data and extends context to 128K–256K. This is not cosmetic: OctoThinker shows mid-training decides whether a base model is even RL-ready — the same RL recipe makes Qwen soar and Llama stall, and reasoning-dense mid-training closes the gap. MAI runs an explicit mid-training stage (STEM up-weight, context to 256K) precisely to “build a strong foundation for reasoning RL”; DeepSeek, Qwen, and MiMo all do their own version (MiMo’s 3-stage mixture, Llama’s high-quality “annealing” tail).

Takeaway. Pre-training is engineering now: ladder-and-EG scaling with deliberate over-training, FP8/FP4 precision, a small stability toolkit (QK-norm, z-loss, careful init), and a mid-training stage that quietly determines the RL ceiling — with the AdamW-vs-Muon optimizer question newly open.

Post-training I: SFT, cold-start, and distillation

Pre- and mid-training give you a base model with broad competence but no idea how to behave — how to follow instructions, reason before answering, or use tools. Post-training fixes that, and it has settled into a two-act structure: a supervised stage that sets the starting point, then reinforcement learning that does the climbing (next section). This section is about the starting point, where the deepest philosophical disagreement in the whole pipeline lives.

What SFT is actually for. It is tempting to think supervised fine-tuning is where capability comes from. In the modern recipe it is mostly a readiness gate: teach the model the chat/tool format and seed it with enough competence to produce some good rollouts, so RL has signal to amplify. Over-do it and you ossify the policy and kill the exploration RL needs; under-do it and RL has nothing to push on. The instruction data itself is increasingly synthetic — the lineage runs from WizardLM’s Evol-Instruct (have an LLM rewrite seed instructions to be harder and more diverse) through persona-driven generation and constraint taxonomies.

The “cold start” and the pure-RL surprise. The most influential post-training result of the era is DeepSeek-R1 (DeepSeek-AI, 2025). Its R1-Zero variant applied GRPO directly to the base model with no SFT at all, rewarded only by rule-based verifiable signals — and reasoning emerged: AIME accuracy climbed from 15.6% to 77.9%, response length grew on its own, and the model spontaneously developed self-checking (the famous “aha moment,” a spike in the word “wait”). The catch was readability and language-mixing, which the full R1 fixes with a small cold-start SFT (a few thousand curated long-CoT examples) before RL. That template — optional cold-start → RL → reject- sample → RL — is now standard (Qwen3, Kimi, MiMo, Magistral, MAI all run a version).

Self-improvement is how you make SFT data at scale. When you can verify answers, you don’t need humans to write reasoning traces — the model writes its own and you keep the good ones. This is one idea in three dresses: STaR (Zelikman et al., 2022) filters self-generated correct rationales; LMSI (Huang et al., 2022) filters by self-consistency with no labels; ReST-EM (Singh et al., 2023) shows the generate→filter→SFT loop is expectation-maximization and beats training on human data whenever you can check correctness. Llama 3’s rejection-sampling and Tulu 3’s pipelines are direct descendants, and related self-correction engines (Self-Refine, Chain-of-Verification) feed the same loop.

Self-distillation as a save-point. A newer, subtler use is keeping a thousand-step RL run alive. MAI leans heavily on self-distillation: periodically SFT a fresh checkpoint on the RL run’s own rollouts, then resume RL. They use it to move from a raw prompt to the chat format, to recover from collapses (resuming from a pre-crash checkpoint fails because the instability was already baked into the weights), and to carry progress onto a new base model. Their ablations are quotable — ~1M traces suffice, incorrect-answer traces work about as well as correct ones, and traces from a range of late checkpoints beat traces from the single final policy. The technique has a clean formalization in on-policy self-distillation (Zhao et al., 2026).

Why does any of this work — why does the same RL help one base model and not another? Because RL mostly amplifies behaviors the base model already has. The “cognitive behaviors” study (Gandhi et al., 2025) shows that verification, backtracking, subgoal-setting, and backward-chaining are present in Qwen and largely absent in Llama, and that priming Llama with those behaviors (even via incorrect but well-structured traces) makes it RL-trainable. This is the deep reason mid-training and cold-start matter: they install the behaviors RL will sharpen.

Divergence — inherit vs learn. The dominant move in 2025 is distillation: R1’s 800K long-CoT traces, SFT’d into small Qwen and Llama models, beat large-scale RL from scratch at the same size — so DeepSeek even distills R1 back into V3’s own SFT data, and most labs distill from a strong reasoner somewhere. MAI takes the opposite stance as a founding principle: “capabilities should be learned, not inherited,” refusing to distill from third-party models because (they argue) imitated intelligence lacks the steerability and robustness needed for long climbs. It is the cleanest philosophical fork in the field: distillation is cheaper and often better per dollar, but only RL can explore beyond any teacher.

Distillation, three ways — and the rise of MOPD

Because distillation is now load-bearing (and because the 2026 reports use very different kinds), it’s worth separating three flavors that get lumped together:

• Off-policy (classic) distillation. The student trains on a fixed set the teacher produced — the teacher’s sampled completions (sequence-level SFT) or its per-token logits on teacher/ground-truth sequences. Simple and cheap, and it’s how small models get their reasoning: DeepSeek-R1’s 800K long-CoT traces SFT’d into small Qwen/Llama, or Gemma 4 matching a larger Gemini teacher’s logit distribution (Gemma Team, 2026). Weakness: exposure bias — the student only ever sees the teacher’s trajectories, never its own, so errors compound at inference (forward-KL, mode-covering).
• On-policy distillation (OPD). The student generates its own rollouts and the teacher scores those student-visited prefixes with dense, token-level guidance (reverse-KL, mode-seeking). This is imitation learning à la DAgger — training on the states you’ll actually visit cuts compounding error from O(T²) to O(T) — closing the train/inference gap that off-policy KD suffers. The idea goes back to GKD (Agarwal et al., 2023) and MiniLLM (Gu et al., 2023), was repackaged as a clean recipe in late 2025, and by 2026 is a standard primitive (there’s already a survey).
• Self-distillation. Teacher and student are the same lineage — a privileged, earlier, or stronger checkpoint of the model itself. MAI’s “save-point” self-distillation (above) is one flavor; Nemotron- Cascade 2 (NVIDIA, 2026) uses the model’s own intermediate checkpoints as teachers to repair regressions during a long RL cascade.

The 2026 consolidation pattern — MOPD. The single most important distillation development is Multi-Teacher On-Policy Distillation: train several independent, per-domain RL specialists (math, code, agentic, instruction-following, chat), then fuse them into one generalist by distilling all of them onto the student’s own rollouts — each teacher supplying a dense token-level reverse-KL signal on the prompts in its domain, alongside a verifiable outcome reward. It decouples development (teams build domain teachers in parallel, with no cross-domain interference) and inherits nearly all of each teacher’s peak skill: MiMo-V2-Flash reports 0.937 vs 0.882 for the best merge/Mix-RL baseline on Qwen3-30B-A3B (Xiaomi, 2026; MOPD, 2026).

Read across the labs and the “specialists → OPD-merge” shape is everywhere — but the knobs differ:

How the 2026 models distill. The dominant new shape is “train domain specialists separately, then merge them into one model via on-policy distillation on the student’s own rollouts.”

Two design rules recur and are worth memorizing:

• Same-origin teachers. OPD is stable when teachers share the student’s tokenizer/vocab and init family. Swap in a stronger external teacher (e.g. Qwen3-235B) and the student’s per-token KL jumps ~5× (0.04 → 0.19), entropy contracts toward a single mode, accuracy drops, and the aggressive top-k form can diverge outright (~step 18). Distill from your own family, not the best model on the board.
• Loss form is a support-overlap trade. Full-vocabulary reverse-KL is most faithful but you must move a ~100k-way distribution per token (DeepSeek pays for it via hidden-state caching); sampled-token/top-k are cheaper and — per Nemotron — can win on agentic tasks, because broad distribution-matching amplifies noise wherever the student’s prefixes fall outside the teacher’s reliable support.

Insight — distillation is eating the “consolidation” step. Two years ago you merged skills by mixing all the RL data together and hoping (Mix-RL); the 2026 default is train specialists separately, then OPD-merge them. It’s more modular, more stable, and — because the reward is a distributional divergence rather than a scalar — meaningfully harder to reward-hack. The catch (and MAI’s objection) still stands: a student can only be pulled toward teachers that already exist, so OPD industrializes inheritance — and only RL still explores past every teacher.

Takeaway. SFT/cold-start sets the starting point and installs RL-ready behaviors; verification turns the model into its own data factory (STaR/ReST-EM/self-distillation); and distillation — off-policy, on-policy, or self — is how 2026 labs consolidate many specialists into one model, with MOPD now the default. The open question is still inherit vs learn — OPD industrializes inheritance, but only RL explores past every teacher.

Post-training II: RL, the engine of reasoning

This is the heart of the modern recipe and the part that changed most in 2024–2026. Supervised learning can only imitate the trajectories in its dataset; reinforcement learning lets the model generate its own attempts and be scored by a reward, which is what makes long-horizon reasoning and tool use trainable. The remarkable thing is how standardized — and how fragile — this stage has become.

From PPO to GRPO

Classic RLHF used PPO (Schulman et al., 2017), whose clipped surrogate objective is still the substrate everything inherits: maximize the reward-weighted probability ratio, but clip the ratio to a trust region so a single update can’t move too far. PPO needs a value model (a second, policy-sized network) to estimate the baseline for advantages — expensive, and awkward when the reward only arrives at the end of a long chain of thought.

GRPO (Shao et al., 2024, DeepSeekMath) is the move that defined the era: delete the value model and estimate the baseline from a group of sampled answers to the same prompt. For a prompt \(q\), sample \(G\) responses, score each with reward \(R_i\), and give every token of response \(i\) the group-relative advantage \(\hat A_i = (R_i - \text{mean}(R_{1..G})) / \text{std}(R_{1..G})\). That’s it — a Monte-Carlo baseline, no critic, and a perfect fit for cheap verifiable rewards. GRPO (or a close sibling) is now the RL backbone of DeepSeek-R1, Qwen3, MiMo, GLM-4.5, Magistral, Nemotron, and MAI-Thinking-1.

Figure 5. PPO’s clipped surrogate is the substrate; GRPO drops the value model for a group-relative baseline; then a wave of fixes (clip-higher, token-level loss, KL removal, adaptive-entropy / outer clip, CISPO) makes it stable for long-CoT and off-policy training. The algorithm is rarely the contribution — the stabilization is.

Consensus (through 2025) — the algorithm is a commodity, the stabilization is the work. Almost every lab adopted a GRPO-family, critic-free, group-relative, token-level objective with verifiable rewards. DeepSeek-R1’s own thesis says it plainly: the keys to reasoning are “hard questions, a reliable verifier, and sufficient compute” — not a clever loss. (As we’ll see, long-horizon agentic RL is now complicating this — including a partial return to the critic.)

Naive GRPO is fragile: the fix-wave

Reproducing R1-scale results revealed that vanilla GRPO collapses, and a wave of fixes followed — almost all about how the surrogate is normalized and clipped:

• DAPO (Yu et al., 2025) is the de-facto “GRPO++”: clip-higher (a looser upper clip so low-probability exploratory tokens can grow — preserving entropy), dynamic sampling (drop prompts with pass-rate 0 or 1, whose group advantage is zero), a token-level loss (normalize over all tokens, not per-sample, killing a length bias), and overlong-reward shaping. It also drops the KL term.
• Magistral (Mistral, 2025) and MiMo confirm the recipe: eliminate KL, length-normalize the loss, clip-higher, filter zero-advantage groups.
• BAPO (Xi et al., 2025) generalizes clip-higher to an adaptive controller for the off-policy regime that asynchronous infra creates.
• MiniMax’s CISPO clips the importance-sampling weight instead of the token update, so no rare reflective token is ever dropped — 2× faster than DAPO.

Divergence — keep or drop the KL term. A genuine split: drop the KL-to-reference for long-CoT reasoning (the policy should move far from init — DAPO, Magistral, MiMo, MiniMax), but keep it for RLHF alignment (stay near a trusted model — Tulu 3, and DeepSeek’s alignment stages). DeepSeek-V3.2 threads the needle with an unbiased KL estimator and very weak KL in math domains.

The long-horizon re-think: GSPO, and the return of the critic

For most of 2024–2025 the story above (“GRPO + a few fixes”) really was the whole story. But through 2026, as labs push from single-turn reasoning into long-horizon agentic RL — agents that run for hours, over dozens of tool calls — that consensus has started to crack in two interesting directions, and this is the most important update to the picture.

Direction 1 — go sequence-level (GSPO). GRPO’s importance ratio is per token, which on MoE models is noisy (a token’s experts can differ between the rollout and training passes) and forces the “router-replay” hack from earlier. Qwen’s Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025) instead defines the importance ratio and clipping at the sequence level (length-normalized), which is more stable, matches the sequence-level reward, and — notably — eliminates the need for routing replay on MoE. Qwen reports GSPO behind the latest Qwen3 models; it is the cleanest “stay critic-free but fix GRPO’s unit of analysis” answer.

Direction 2 — bring back the critic (SAO / PPO). The sharper reversal comes from GLM. The slime-trained GLM line (GLM-5, Zhipu, 2026) starts on GRPO (plus an “IcePop” train/inference-mismatch fix), but the later GLM-5.2 (750B-A40B) explicitly abandons group-relative optimization for a critic-based method in its long-horizon stage. The reason is concrete and worth internalizing: when a very long agent trajectory is compacted into multiple sub-traces, different rollouts of the same prompt yield different numbers of trainable traces with wildly different lengths — so GRPO’s “compare a clean group of comparable rollouts” assumption breaks. A critic estimates token-level advantages for a single rollout, with no requirement that rollouts be group-comparable, which fits compaction naturally.

The concrete algorithm behind GLM-5.2’s agentic climb, SAO — Single-rollout Asynchronous Optimization (Hou et al., 2026) — makes the trade explicit: replace group sampling with one rollout per prompt, train a real value model (with faster critic updates and a frozen-attention scheme so the MoE projections train while attention is held fixed), and add a strict double-sided token-level clip (DIS) that masks divergent gradients. The payoff is stability: vanilla GRPO collapses at ~160 asynchronous steps, while SAO trains stably for ~1,000 steps and beats GRPO-family baselines on SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. After three years of everyone deleting the value model, the value model is back — for the long-horizon, asynchronous case. GLM-5.2 then consolidates the way DeepSeek-V4 does: train separate domain specialists, then merge them into one generalist via on-policy distillation (Z.AI reports fusing 1,000+ specialists in ~2 days on slime).

Concretely, SAO is four moves layered on the PPO/GRPO surrogate:

1. Single-rollout sampling. One rollout per prompt instead of a group. In asynchronous RL a group is only as fast as its slowest member — training must wait for the laggard — so group sampling is latency-driven off-policy and doesn’t fit online/continually-evolving environments. One rollout removes that barrier (and, the authors argue, generalizes better).
2. A critic — made affordable. Drop the group baseline and estimate token-level advantages with a value model. To pay for it, SAO updates the critic more often than the actor and fine-tunes the value model with frozen attention (train only its MoE/projection params), so the critic is cheaper to fit than a full second policy.
3. Skip-observation token-level GAE. For multi-turn agent traces with interleaved tool/environment feedback, compute advantages across action-to-action boundaries and skip the observation tokens the model didn’t generate — so environment text doesn’t inject noise into the value estimate.
4. Strict double-sided token-level clipping (DIS). Clip both sides of the token ratio to mask divergent gradients — a simpler cousin of the “IcePop” mask that keeps off-policy updates bounded.

The numbers (on a Qwen3-30B-A3B base): SWE-Bench Verified 23.0 → 27.0 (GRPO) → 29.8 (SAO); AIME 2025 80.4 → 84.2 → 97.3; BeyondAIME 53.3 → 54.8 → 74.8 — and, crucially, vanilla GRPO collapses at ~160 asynchronous steps while SAO stays stable to ~1,000.

Open question — is the critic worth its cost, and does this generalize? Read SAO skeptically on two axes. (1) Cost: it re-introduces exactly what GRPO deleted — a second, ~policy-scale value network — and the “frozen-attention, faster-critic” tricks reduce but don’t erase that overhead; the bet is that long-horizon agentic RL needs token-level credit assignment badly enough to pay for it. (2) Scale evidence: the controlled ablations are at 30B (Qwen3-30B-A3B), so the clean “SAO > GRPO” curves are a small-model result; the only large-scale evidence is a single production deployment at 750B (GLM-5.2), not a controlled 750B A/B test. That’s a genuinely strong signal — a frontier lab shipped it — but “ablate at 30B, deploy at 750B” is itself the leap-of-faith the next section is about. Community acceptance is early: one lab’s production use plus a fresh paper, against a field that spent two years learning to love being critic-free.

Divergence — the algorithm is becoming task-specific again. The clean 2025 narrative (“GRPO won, the algorithm is a commodity”) is giving way to a 2026 one: GRPO/CISPO for short, verifiable tasks; GSPO for stable MoE RL; critic-based PPO for long, compacted, agentic trajectories. GLM-5.2’s return to PPO is the headline, but the deeper point is that trajectory length and structure now drive the choice of RL algorithm. Note DeepSeek-V4 (2026) takes yet another route — keep GRPO per-domain-expert, then fuse the experts with on-policy distillation — and MiniMax’s M2 (2026) builds a whole agent-native RL system (“Forge”) around long, uneven trajectories. There is no longer a single default.

The entropy problem

The single most important failure mode is entropy. Too little and the policy collapses to a deterministic, un-exploring model that saturates; too much and it spews gibberish and runaway length. The Entropy Mechanism study shows these are one phenomenon: \(\Delta H \propto -\text{Cov}(\log \pi, \text{advantage})\), and the fixed PPO clip systematically removes the entropy-increasing updates → monotone collapse, with a predictable ceiling (\(R = -a\,e^{H} + b\)). The fixes span where you intervene: at the clip (DAPO’s clip-higher; BAPO’s adaptive bounds; MAI’s adaptive entropy control — an integral controller that nudges the upper-clip relaxation toward a target entropy \(H^\star=0.3\)), at the covariance level (Clip-Cov / KL-Cov suppress the specific high-covariance tokens), or via an entropy bonus — which both the Entropy-Mechanism authors and MAI report underperforms the adaptive approaches.

Trick — entropy as a control loop. MAI’s contribution here is to treat entropy like a thermostat: measure it each step and adjust the clip width to hit a setpoint, rather than tuning a fixed bonus. It also adds a hard outer clip (\(r_{max}=50\)) on all branches to kill the catastrophic gradient spikes that GRPO’s deliberately-unclipped branches can cause.

Reward design: why verifiable rewards won

The reward is where RL succeeds or fails, and there are three sources, each hackable:

Figure 6. Three reward sources trade breadth for robustness. Verifiable rewards (math answer-checkers, code unit tests) are narrow but hard to game; reward models and LLM-judges cover open-ended tasks but get hacked. Non-negotiable objectives like safety are gated rather than averaged in.

• Verifiable rewards — does the math answer match, do the unit tests pass — are cheap, low-noise, and hard to game at the reward level. This is why math and code dominate RL, and why Tulu 3 formalized RLVR (“the policy is only rewarded when verifiably correct,” with the blunt finding that verifiable-only beats reward-model + verifiable — the RM just adds hackable noise, Lambert et al., 2024). DeepSeek-R1 deliberately avoids neural reward models for exactly this reason.
• Reward models hack prompt-independent artifacts (length, markdown, emojis); robust-RM training via counterfactual augmentation (Liu et al., 2024) helps.
• LLM-as-judge is convenient for open-ended tasks but catastrophically foolable: a single meaningless token (“Solution”, “:”) can elicit a false “correct” up to ~80% of the time, even from frontier judges (Zhao et al., 2025).

So labs compose rewards, and the composition matters. MAI uses a decomposed reward \(R = R_{task} + w_{lang}R_{lang} - w_{len}R_{len}\) — adding a language-consistency reward (mixed- language CoTs destabilize training) and a difficulty-aware length penalty. And for non-negotiable objectives it gates rather than averages: an unsafe response gets the minimum reward regardless of quality (motivated by the finding that 87.8% of policy-non-compliant responses still scored ≥3 on the reward model — averaging would let quality buy back safety). We return to this in Alignment.

Difficulty ≠ trainability

A subtle but universal filter: which prompts are worth training on? Not the hardest — the learnable ones. For a binary outcome reward, the learning signal is the reward variance \(\hat p(1-\hat p)\), which is maximized at a 50% success rate and zero at both extremes: a prompt the policy always fails or always solves produces identical rewards across the group, so the group-relative advantage — and the gradient — is exactly zero.

Figure 7. A prompt only teaches when its outcome is uncertain. The learning signal is the reward variance \(p(1-p)\): “too hard” and “too easy” fail for the same reason. Labs filter to a middle band.

Every lab implements this. MAI filters prompts to a pass-rate band of [0.1, 0.8] (with an early-exit pre-filter to save rollout cost); DAPO’s dynamic sampling drops the {0,1} groups; MiMo keeps an easy-data pool to stabilize updates as more problems are mastered. This is the same idea that drives environment curricula in the agentic-RL world — explored at length in the companion post on environment scaling.

Spending tokens wisely, and agentic RL

Reasoning models “overthink,” so labs add difficulty-aware length penalties — penalize length in proportion to how easily a prompt is solved, so hard problems can think longer (Xiang et al., 2025; MAI adopts exactly this). Qwen3 exposes it to users as a “thinking budget.” And the frontier is agentic, multi-turn RL: the same GRPO objective over a trajectory of model-and-environment steps, run inside sandboxed containers with verifiable rewards (tests passing, a database reaching a target state). MAI’s “agentic climb” builds SWE environments from 102M GitHub PRs filtered to ~4.87M with linked issues, graded by fail-to-pass / pass-to-pass tests inside its on-demand sandbox; DeepSeek-V3.2 and Kimi K2 synthesize thousands of tool environments. The supply of these environments is its own deep topic — again, the environment-scaling post.

The unglamorous stability tricks

Finally, the fixes that don’t make the abstract but make the run converge: top-p mask replay (reuse the sampling truncation mask at train time so train and inference agree — MAI, DeepSeek-V3.2); MoE router replay (an MoE picks different experts in the inference vs training engine, an off-policy gap baked into the architecture — fixed by replaying routing, Ma et al., 2025; DeepSeek’s “Keep Routing”); an FP32 LM head to fix train/inference precision mismatch (MiniMax); async-RL staleness bounds; and self-distillation as a numerical save-point (previous section). Most labs build on (or replace) open RL frameworks like verl/HybridFlow and OpenRLHF — MAI wrote its own (“Rocket”) precisely because those didn’t scale to thousand-GPU asynchronous RL.

Table 2. The post-training/RL recipe: heavily converged on a GRPO-family verifiable-reward backbone, with real divergence on how much RL (vs DPO), whether to keep KL, and how rewards are composed.

Divergence — how much RL at all? Not everyone is in the RL-heavy camp. Llama 3, Gemma 2, and Hunyuan-Large deliberately rely on DPO / rejection-sampling / distillation and keep RL light or absent (Llama’s stated thesis is complexity management). DeepSeek, MAI, MiMo, and MiniMax bet the other way and pour compute into RL (DeepSeek-V3.2 now spends >10% of pre-training cost on RL, and rising). This — not the choice of GRPO variant — is the consequential fork.

Takeaway. RL is now a standardized but fragile engine: a GRPO-family, verifiable-reward, token-level objective, where the real work is reward design, entropy control, trainability filtering, and a pile of train/inference-consistency tricks. The big bets are how much to lean on RL and how to compose rewards.

Agentic RL: credit assignment over long horizons

The last section’s algorithms were mostly proven on single-turn reasoning — one prompt, one long chain of thought, one verifiable reward at the end. Agentic tasks break that mold: a coding or research agent runs for dozens to hundreds of turns, calling tools and reacting to a stochastic environment, and the only trustworthy reward often arrives once, at the very end. A recent Stanford survey (Zhang et al., 2026) frames this as two distinct regimes of the credit-assignment (CA) problem:

• Reasoning RL — spread credit across tokens/steps inside a single CoT (500–30K tokens), deterministic environment. This is where GRPO-and-fixes matured; CA is “solved enough” by a group-relative baseline.
• Agentic RL — 10–100+ turns, each an LLM call plus an environment interaction, totaling 100K–1M+ tokens, with stochastic transitions and partial observability. Here episode-level reward becomes almost uninformative: in one SWE-bench setup an agent averaged ~64 turns and ~131K tokens, and a single wrong tool call at turn 3 gets the same penalty as dozens of correct actions afterward.

(For the full design space beyond credit assignment — environments, tool integration, memory, multi-agent — see the broader survey of agentic RL for LLMs.)

Why agentic RL is genuinely different. The survey’s headline is that agentic CA is not just “reasoning RL, but longer.” It forces techniques with no real precedent in single-turn RL — turn-level MDP reformulations, hindsight/counterfactual credit, and privileged asymmetric critics — and it re-opens questions (critic vs critic-free, dense vs sparse reward) that the reasoning-RL era thought it had closed. This is the live frontier of 2026 post-training, and it’s why the “return of the critic” (SAO, last section) happened here first.

Credit assignment: from one scalar to per-turn (and per-token)

The core question is how to spread one terminal reward over a long trajectory. The methods form a clean spectrum from coarse to fine:

• Episode/outcome-level (the default). Broadcast the trajectory’s scalar reward to every token, GRPO-style. Simple, and what most shipped frontier models still do — but the noisiest possible credit at long horizons.
• Turn-level, critic-free. Keep GRPO’s group baseline but compute advantages per turn. MT-GRPO (Wei et al., 2025) estimates turn-level advantages that drop into GRPO; GiGPO (Feng et al., 2025) adds a clever anchor-state grouping — it finds repeated environment states across a group of trajectories and compares the actions taken from each, producing step-level “micro-advantages” at the same memory and rollout cost as GRPO (+12% ALFWorld, +9% WebShop). Tree-structured variants share trajectory prefixes so a single outcome reward induces step-wise signals.
• Turn-level, with a critic. Bring back a value model, but at the turn granularity. ArCHer (Zhou et al., 2024) is the template: a high-level off-policy critic learns a turn-level Q-function (“which turn mattered”) while a low-level on-policy actor optimizes tokens within a turn (“which tokens mattered”) — ~100× more sample-efficient than on-policy baselines. GLM-5.2’s SAO (last section) is the production-scale cousin: a single-rollout token-level critic that survives trajectory compaction, exactly where group-relative methods break.
• New-to-agentic. Hindsight relabeling (reinterpret a failed run as a success for whatever it did achieve) and privileged/asymmetric critics (the critic sees state the actor can’t) are, per the survey, the genuinely novel agentic contributions.

Consensus — algorithmic CA is winning over learned step-rewards. The striking thing about the shipped models is how few use an explicit per-step reward model. The frontier answer to long-horizon credit is better advantage estimation (turn-level grouping, anchor states, compaction-aware critics), not a separate network scoring every step. The fine-grained methods above are still mostly research-scale (1.5B–7B); production leans on outcome/verifiable rewards plus smart baselines.

Stepwise reward: process rewards, and why the frontier is wary

The other half of the problem is evaluating intermediate steps at all. The lineage is well-developed for math:

• Process reward models (PRMs). “Let’s Verify Step by Step” (Lightman et al., 2023) showed step-level human labels beat outcome labels for math verification; Math-Shepherd (Wang et al., 2023) removed the humans, auto-labeling a step by the empirical fraction of Monte-Carlo rollouts from it that reach the correct answer, then training step-by-step PPO. Generative PRMs (score with a rationale) are the current refinement.
• Agentic PRMs. Porting this to agents means scoring actions: AgentPRM (2025) rewards each step by its “promise and progress” toward the goal, using TD + GAE to train a PRM that serves as the critic; SWE-/Code-PRMs add execution feedback on partial code edits.

But here is the field’s sharpest agentic-reward lesson, and it’s a negative result:

Divergence — the frontier mostly rejected PRMs. DeepSeek-R1 (2025) tried both process reward models and MCTS and abandoned them: neural PRMs invite reward hacking (the policy games the scorer), “a step” is ambiguous to define in open-ended reasoning, and PRMs need constant costly re-labeling; MCTS’s token-level branching factor is simply too large. Their conclusion — internalize search into the weights with outcome/verifiable rewards — became the default. Kimi K2 (2025) shows the pragmatic middle path for non-verifiable agentic tasks: a rubric-based self-critique reward (the model grades itself against explicit rubrics), with the critic continuously recalibrated by verifiable rewards. So the frontier’s “stepwise reward” is usually an LLM-judge with a rubric — not a trained PRM.

How the labs actually train agents

Reading the reports, the agentic recipe is less about a single algorithm and more about environments + rollout infrastructure + reward plumbing:

• Kimi K2 — the most agentic-first open report. It manufactures its data: start from ~3,000 real MCP tools, evolve to 20,000+ synthetic tools, spawn diverse agents and rubric-graded tasks, simulate multi-turn trajectories (with real execution sandboxes for code), and keep only trajectories an LLM judge passes. RL then combines RLVR (tests/math) with the rubric self-critique; the infra uses partial rollouts — pause an unfinished long trajectory and resume it next iteration — so a few 100K-token episodes can’t stall the batch.
• GLM-4.5 → 5.2 — agentic-optimized (tool use, web, coding) on the slime async infra; GLM-5.2 adds trajectory compaction and the SAO critic for hour-long tasks, then OPD-merges specialists (previous sections).
• MiMo-V2-Flash — large-scale agentic RL, consolidated via MOPD, with MTP to speed up the many rollouts agentic RL demands.
• Qwen3 — tool-integrated/agentic RL stabilized by GSPO (sequence-level ratios suit multi-turn MoE rollouts).
• DeepSeek — the outcome-reward purist (above); V3.2/V4 extend long-context and agentic use but keep rule-based/verifiable rewards.
• Grok 4.5 — mid-trained on real Cursor developer-session traces (the Data section): arguably the highest-signal agentic data of all, recordings of humans actually doing the target task.
• Nemotron — agentic reasoning via multi-environment RLVR plus MOPD consolidation.

The through-line is that the environment is the bottleneck — the entire subject of a companion post, Environment Scaling for Agentic RL. Once you can simulate enough realistic, verifiable, resettable tasks, credit assignment and reward design are what turn those environments into capability.

The credit-assignment and stepwise-reward toolbox. Finer granularity buys sample-efficiency but costs compute and invites reward hacking — which is why shipped frontier models cluster at the coarse, robust end (outcome + verifiable rewards) and add fine-grained credit through advantage-estimation tricks, not trained step-reward networks.

Takeaway. Agentic RL is where 2026’s action is: horizons of 100K–1M tokens make one terminal reward almost useless, so labs sharpen credit assignment (turn-level grouping like GiGPO, compaction-aware critics like ArCHer/SAO) far more than they add stepwise reward models — which DeepSeek-R1’s public post-mortem showed are reward-hackable and hard to define. In practice the winning ingredients are realistic environments, partial/async rollouts, verifiable-or-rubric rewards, and advantage estimation at the right granularity — with the environment itself the real bottleneck.

Catastrophic forgetting and continual learning

Every stage in this post overwrites the one before it. Mid-training moves the base model, SFT moves it again, each RL climb moves it further, and an agent that keeps training on freshly captured experience moves it forever. The risk is catastrophic forgetting (CF): teaching the model something new silently erases something old (McCloskey & Cohen, 1989; Kirkpatrick et al., 2017). This is not a corner case — it is a central tension of the whole pipeline, and the 2024–2026 reports increasingly treat post-training as a continual-learning problem: how to establish behavior, refine it on learner-generated states, and preserve it across stage transitions (survey; unified view).

Where it bites, concretely:

• Continual pre-training on a new domain erodes general ability.
• SFT / alignment pays an “alignment tax” — safety and instruction-tuning gradients interfere with the parameter subspaces that hold general skills, a CF-like effect (Sun et al., 2026).
• Multi-stage / Cascade RL — training math, then code, then agentic in sequence means each stage erodes the last (exactly why Nemotron-Cascade uses OPD to “recover benchmark regressions”).
• Agentic continual learning — the case you raised: an agent that keeps learning from new experience risks trading yesterday’s competence for today’s skill.

RL’s Razor: forgetting is a KL problem

The sharpest recent result reframes all of this. RL’s Razor (Shenfeld et al., 2025) shows that, at matched new-task accuracy, RL forgets less than SFT — and that the culprit is not the algorithm but the KL divergence from the base model, measured on the new task. On-policy RL is implicitly biased toward the KL-minimal solution among the many that solve a task; SFT can drift arbitrarily far. (The tell: toggling GRPO’s negative gradients in or out doesn’t change forgetting; swapping on-policy data for off-policy does.) This retroactively explains the whole “keep the KL term” camp from the RL section: the KL penalty is, mechanically, a forgetting knob — and it explains why the on-policy family (RL, on-policy distillation) is structurally gentler on old skills than plain SFT.

Insight — stay on-policy, stay close. If forgetting tracks KL-on-the-new-task, the levers are obvious: prefer on-policy updates, keep an explicit KL leash to a trusted reference, and don’t push any single stage further from the base than the new capability requires. Much of the toolbox below is just different ways of bounding that same KL.

The mitigation toolbox

Five families, and the frontier reports combine all of them:

• Replay / rehearsal. Mix old data back in. The canonical instance is InstructGPT’s PPO-ptx (Ouyang et al., 2022) — blend pretraining gradients into RLHF to cancel the alignment tax; Kimi K2’s PTX auxiliary loss is the same idea in a 2026 flagship. Newer variants schedule replay by a model-centric clock — replay when the weights have actually moved, following the Ebbinghaus forgetting curve (MSSR, 2026).
• Proximity regularization. Stay near a reference: the RLHF KL penalty, classic EWC (Fisher-weighted anchoring of important weights, Kirkpatrick et al., 2017), or the newer orthogonal gradient projection that strips out exactly the part of an alignment update that would collide with general-capability directions (Sun et al., 2026).
• Parameter isolation (PEFT). Freeze the base and learn adapters (LoRA); you can’t overwrite what you don’t train — at the cost of capacity.
• Model merging / souping. Average the fine-tuned model back toward the base or toward sibling checkpoints: model soups (Wortsman et al., 2022), task arithmetic (Ilharco et al., 2022), and the checkpoint-souping Llama 3 runs across its RM/SFT/DPO models. Cheap, post-hoc, and surprisingly good at buying back general performance.
• Distillation-based recovery. The family most relevant to your question. Self-distillation (MAI’s save-point trick, §SFT) and especially on-policy distillation can re-invoke capabilities lost during fine-tuning: use an earlier checkpoint as a fixed teacher and OPD the student back toward it on-policy — because it stays on-policy against a fixed target, it converges to the teacher’s behavior without the regression SFT causes (Thinking Machines Lab showed OPD recovering nearly full IF-eval after document fine-tuning with no knowledge loss; when it stalls, an off-policy cold start fixes it, Li et al., 2026). This is why MOPD-style consolidation doubles as forgetting-repair: Nemotron-Cascade OPDs from the strongest intermediate teacher to undo Cascade-RL regressions, and the general “specialists → OPD-merge” pattern (GLM-5.2, DeepSeek-V4, MiMo-V2) decouples skills so learning one can’t clobber another.

The anti-forgetting toolbox. Note the RL’s-Razor throughline: replay, KL penalties, and on-policy distillation all work by keeping the model’s distribution close to where it started; merging does the same thing after the fact.

The agentic twist: learn in memory, not weights

Your specific worry — an agent that forgets as it accumulates experience — has provoked a distinct answer: move continual learning out of the weights and into context/memory. Rather than fine-tune on every new experience (and risk CF), 2026 agent systems increasingly expose memory as tools the agent learns to use — store, retrieve, summarize, discard — while the base weights stay frozen: AgeMem (Yu et al., 2026) trains exactly these operations with step-wise GRPO, and neurocognitive designs add a consolidation loop that replays high-utility traces and prunes the rest. The base model stops being the thing that remembers; a bounded, curated memory does. When labs do update weights on new experience, they fall back on the toolbox above — replay, a KL leash, OPD-recovery — plus the “specialists → merge” decoupling. (The environments that generate all this experience are the subject of the companion post, Environment Scaling for Agentic RL.)

How the frontier labs handle it

• OpenAI (InstructGPT) — introduced PPO-ptx replay to cancel the alignment tax; the template everyone inherited.
• Kimi K2 — PTX auxiliary loss + temperature decay to hold onto pretraining knowledge through heavy agentic RL.
• MAI — self-distillation as a save-point: resume from a collapse, and carry progress onto a new base without retraining from scratch.
• DeepSeek-R1 — a final RL stage over all scenarios to restore general helpfulness after reasoning-focused RL, with a deliberately short model-preference-reward phase (the last 400 of 1,700 steps) to avoid over-drift and reward hacking.
• Nemotron-Cascade / -3 Ultra — on-policy distillation from the best intermediate checkpoint as an explicit regression-repair device throughout Cascade RL.
• GLM-5.2 · DeepSeek-V4 · MiMo-V2 — build domain specialists separately, then OPD-merge — the most scalable anti-forgetting move, because the skills are never entangled in the first place.
• Llama 3 — data mixing plus model averaging/souping across checkpoints.
• Qwen3 — fuse thinking / non-thinking into one model so neither mode is forgotten.

Open question — true lifelong learning is still unsolved. Every technique here bounds forgetting; none abolishes it. No frontier model yet updates its own weights online, in production, from a stream of new experience without a careful offline replay / merge / distill cycle — which is precisely why the agentic world is routing around the problem with external memory. Weight-space lifelong learning remains the open frontier.

Takeaway. Catastrophic forgetting is the tax on every post-training step, and 2026’s understanding crystallized around one idea — RL’s Razor: forgetting tracks KL-from-base on the new task, so on-policy updates forget less. The working stack is replay (ptx) + a KL leash + model merging + distillation to recover (self- and on-policy), with the deepest structural fix being “train specialists separately, then OPD-merge.” For agents, the pragmatic escape is to learn in memory, not weights — because genuine online, weight-level lifelong learning is still unsolved.

Alignment: helpfulness, safety, honesty

Alignment used to be a final RLHF coat of paint. In the 2026 recipe it is its own set of RL “climbs” with a dedicated reward stack, run alongside the reasoning RL. The framing everyone now shares is a tension to optimize, not a filter to apply: a model must be helpful (comply) and safe (sometimes refuse) at once, and the art is getting both. MAI states the target as “helpful responses that remain compliant with policy”; OpenAI frames the same goal as moving from refusal to safe completion.

The reward stack here is the most heterogeneous in the pipeline, because the objectives (“is this helpful? honest? appropriately styled?”) resist verification. MAI’s helpfulness-and-safety climb combines a reward model trained on human preferences (with reward-hacking mitigations), AI judges (fast, rubric-guided, easy to retarget), and verifiable rewards wherever a constraint is checkable (e.g., “answer in under 10 words”) — the last specifically because verifiable signals are less hackable and stabilize the others.

Trick — gate safety, don’t average it. The single most transferable idea here: some objectives are non-negotiable, and a weighted sum lets a well-written answer buy back an unsafe one. MAI uses lexicographic / gated aggregation — a safety-noncompliant response gets the minimum reward regardless of its other scores — motivated by a damning audit: 87.8% of policy-noncompliant responses still scored ≥3 on the reward model. Averaging would have rewarded them.

Instruction hierarchy. Production models must rank instructions by privilege — system > developer > user > tool output — so an injected “ignore your instructions” in a web page can’t override the system prompt (Wallace et al., 2024, the basis of OpenAI’s Model Spec and gpt-oss’s harmony format). MAI trains this explicitly with adversarial system/developer/user conflicts; it is now a standard safety-SFT/RL ingredient.

Refusals → safe-completions. The clearest alignment evolution is OpenAI’s shift, documented in the GPT-5 system card, from binary hard refusals to output-centric safe-completions: maximize helpfulness subject to the safety policy, which is strictly better for dual-use questions where a high-level answer is fine but operational detail isn’t. gpt-oss adds deliberative alignment (the model reasons over the safety policy at inference). MAI’s safety climb, with its harmful-vs-borderline taxonomy and its explicit fight against over-refusal, is the same philosophy under a different name.

Honesty and calibration. A subtler alignment axis, and one most labs under-treat: a model should answer when it knows and hedge when it doesn’t — without over-hedging into uselessness. MAI’s honesty reward grades responses into five buckets (confident-correct → confident-incorrect), rewarding confident-correct most, penalizing confident hallucination most, and giving abstention a neutral score — explicitly discouraging over-hedging. This connects to a deeper problem (calibration, abstention, and uncertainty in long-horizon agents) that has its own companion post. Some labs now make calibration a headline design goal: Thinking Machines’ open Inkling (975B/41B, natively multimodal) is explicitly trained “to give calibrated answers, including expressing uncertainty rather than guessing” — a sign that knowing what it doesn’t know is graduating from a research topic into a shipped model property.

Divergence — how much to disclose. The methods are converging, but disclosure is not. OpenAI’s system cards are the eval/safety reference (Preparedness categories, red-team hours, safe-completions), yet reveal almost nothing about training; the open recipes (OLMo 2, Tulu 3, Magistral) disclose training in full but have thin safety sections. MAI sits in between, borrowing OpenAI’s safety grammar while disclosing far more of its recipe.

Takeaway. Alignment is now an RL objective with its own composed reward stack, defined by the helpfulness↔safety tension. The portable lessons: gate non-negotiable objectives instead of averaging them, train an explicit instruction hierarchy, prefer safe-completions over hard refusals, and reward calibrated honesty rather than blanket hedging.

Evaluation: measuring the climb

A “hill-climbing machine” can only climb a hill it can measure, which makes evaluation the quiet bottleneck of the whole pipeline. The reports reveal two different evaluation regimes: a cheap, robust one for development (thousands of decisions), and an expensive, public one for release.

For development, loss beats accuracy. MAI makes the strongest version of this argument: for the ~40-benchmark suite it uses to make pretraining and data-mixture decisions, it scores by NLL (loss), not accuracy. The reasons are operational and decisive — accuracy evals need expensive autoregressive generation and often a judge model; multiple-choice ability “emerges” only at large scale and so is noisy early; MATH needs exact \boxed{} formatting and MBPP trips on \n vs \r\n. NLL is the same teacher-forced next-token objective as training, so it is cheap and high-signal — a conclusion echoed by the Signal-and-Noise framework. The fully-open labs build dedicated dev suites (Ai2’s OLMES) for the same reason.

For release, the benchmark zoo. The public scorecards have standardized around a recognizable set: math (AIME, MATH, HMMT), science (GPQA, Humanity’s Last Exam), code (LiveCodeBench, SWE-bench and the harder SWE-bench Pro, Terminal-Bench), knowledge (MMLU and MMLU-Pro), factuality (SimpleQA, FActScore), long-context (RULER, LongBench v2, Michelangelo), agentic tool-use (τ²-bench, BFCL), and increasingly domain suites like HealthBench and MedXpertQA. The MAI report’s headline numbers (52.8% SWE-Bench Pro, 97.0% AIME 2025) live here, as do every other lab’s — but cross-report comparisons should be read with care, because harnesses, prompts, and tool access differ (one reason “tiny” curated subsets and uncertainty-aware scoring are gaining traction, e.g. tinyBenchmarks).

Open question — evaluation is the real bottleneck. As models saturate old benchmarks, the signal moves to a handful of hard, leak-prone sets. The honest labs increasingly rely on private held-out benchmarks (MAI builds its own; this is the only reliable defense against contamination), on verified re-releases of saturating sets (SimpleQA Verified), and on live evaluation that post-dates training (MathArena scoring fresh competitions). You can only climb what you can measure, and the measuring sticks are wearing out faster than we can make new ones.

Contamination is the crisis underneath. If a benchmark has leaked into training, your number is fiction — and MAI notes a telltale symptom: contamination can make a “coding” dataset mysteriously improve unrelated trivia. The countermeasures (20-gram fuzzy dedup, repo/temporal exclusion, removing HuggingFace mirrors) are imperfect, which is why private and live benchmarks are becoming the only trusted yardsticks.

Beyond benchmarks. Because automatable metrics miss style, helpfulness, and safety, labs add human side-by-side evaluations (MAI reports these prominently) and lean on LLM-as-judge — whose own reliability is now benchmarked (RewardBench, JudgeBench) precisely because, as we saw, judges are hackable.

Takeaway. Use cheap, robust NLL to make thousands of development decisions; use the public benchmark zoo plus human evals to report; and treat contamination as a first-class threat — private and live benchmarks are the only durable defense.

Safety and red-teaming

Alignment training (§ above) is the capability to be safe; this section is how labs measure and stress-test it before release. The grammar here was largely set by OpenAI’s Preparedness Framework and is now imitated everywhere: define a small set of tracked risk categories (GPT-4o: cyber, CBRN, persuasion, model-autonomy; GPT-5 and gpt-oss: bio/chem, cyber, AI self-improvement), assign capability levels, and gate deployment on a post-mitigation threshold reviewed by a safety board. Google’s Frontier Safety Framework and Anthropic’s RSP are siblings; MAI’s safety section borrows the same scaffolding.

Red-teaming is continuous and increasingly automated. It runs throughout development, not at the end. MAI sources adversarial prompts from human red-teams plus automated attack frameworks — PyRIT — and jailbreak methods like PAP (persuasion), Crescendo (multi-turn escalation), Tree of Attacks, and even multilingual jailbreaks. OpenAI reports red-teaming in hours and tester counts (GPT-5: 5,000+ hours, 400+ testers). Llama 3 ships an actual safeguard model, Llama Guard, alongside the LLM — safety as a system, not just a model.

Dangerous-capability evals target the catastrophic tail: bioweapon uplift (the recurring five-stage bio taxonomy), cyber-offense (CyberSecEval, CTF challenges), hazardous knowledge (WMDP, paired with unlearning), regulation-derived risk suites (AIR-Bench), and dedicated frontier dangerous-capability evals. gpt-oss adds a methodology specific to open weights: because anyone can fine-tune a released model, OpenAI built an adversarially fine-tuned “worst-case” version (helpful-only RL + capability maximization) and had external groups confirm it still didn’t cross High thresholds — the emerging template for responsible open-weight release.

The reasoning era adds a new safety lever: CoT monitoring. Because reasoning models think in legible chains of thought, you can monitor that reasoning for deception or misbehavior — GPT-5 reports halving flagged-deception rates (4.8%→2.1%) with a CoT monitor. But there’s a catch the field is actively worried about: if you train against the CoT monitor, the model learns to obfuscate its reasoning rather than behave (Baker et al., 2025; Guan et al., 2025). This makes “keep the chain of thought monitorable” a live design constraint for every lab training long-CoT models — MAI, DeepSeek, Magistral included.

The trend — safety shifts left. Across the reports, safety is no longer a release gate; it is woven through the pipeline: PII/CBRN data filtering (pre-training), the reward stack and instruction hierarchy (post-training), CoT monitoring (inference), and red-team + preparedness (release).

Takeaway. Safety has become a pipeline-wide process with a shared vocabulary (preparedness categories, automated red-teaming, dangerous-capability evals), and the reasoning era adds CoT monitoring — useful, but only as long as we don’t train models to hide their thoughts.

The convergent recipe

Step back from the stages and the claim from the top of the post holds up: by 2026 there is one recipe, and the reports are variations on it. Here is the whole pipeline in one breath — curate and dedup human-plus-synthetic data and choose the mixture by scaling-law forecasting; pre-train a RoPE/GQA/SwiGLU/RMSNorm MoE on a tokens-per-parameter ladder, deliberately over-trained, in FP8; mid-train on reasoning-dense data and extend context; SFT/cold-start to install behaviors and format; run GRPO-family RL with verifiable rewards, entropy control, and trainability filtering; align with a gated reward stack and an instruction hierarchy; measure with cheap NLL plus a contamination-guarded benchmark zoo; and red-team against a preparedness framework. If you internalize that sentence, every report in the table below reads like a fill-in-the-blanks.

Table 3. One recipe, many fills. Reading across the columns, the consensus (MoE + modern block + SFT→RL + verifiable rewards) is visible — and so are the handful of real bets (dense vs MoE, AdamW vs Muon, GRPO vs GSPO vs SAO/PPO, RL-heavy vs DPO, synthetic vs human). The mid-2026 wave (DeepSeek-V4, GLM-5.2, Kimi K3, Inkling, Grok 4.5, Qwen3.5) pushes the frontier toward 1M context, linear/hybrid attention, Muon, and long-horizon agentic RL — and, per the Artificial Analysis Intelligence Index, six labs now field a frontier-class model where two did in early June.

What everyone agrees on (the eight points of consensus): the modern decoder block; fine-grained + shared-expert MoE balanced over the global batch; heavy dedup + scaling-law data mixing + a mid-training tail; deliberate over-training; SFT/cold-start → RL; GRPO-family verifiable-reward RL with entropy control and trainability filtering; multi-stage post-training with distillation somewhere; and a safety reward-stack + preparedness/red-team process.

Where the real bets are (the divergences worth arguing about): synthetic vs human data; inherit (distill) vs learn (RL); AdamW vs Muon (which, as of 2026, Muon is decisively winning — Kimi, GLM, and now DeepSeek-V4); aux-loss vs aux-loss-free vs global-batch balancing; dense vs MoE vs hybrid; how much RL vs DPO; the RL algorithm itself — GRPO vs GSPO (sequence-level) vs a return to critic-based PPO for long-horizon agents (GLM-5.2), which is the freshest and most telling shift; keep vs drop the KL term; refusals vs safe-completions; and how much to disclose (fully-open recipe vs benchmark-table-only card).

Takeaway. Read one report deeply and you have read them all — modulo about nine knobs. Those knobs, not the skeleton, are where the interesting disagreements (and probably the next advances) live.

Open challenges

The recipe works, but several of its load-bearing assumptions are shakier than the leaderboard numbers suggest. Here is where I’d point a skeptical eye.

Verification is the ceiling on RL. The entire RL stage rests on rewards you can trust, and we saw that reward models hack artifacts and LLM-judges fall to one-token “master keys.” That is why RL works so well in math and code and so poorly everywhere else — those are the domains with cheap, robust verifiers. Extending reliable verification to genuinely non-verifiable goals (“is this analysis good?”) is the open problem under most of the others.

The data wall meets the synthetic-data dilemma. Deliberate over-training and ever-larger token budgets are running into the finite supply of high-quality human text. The escape hatch is synthetic data — but that is exactly the bet MAI refuses, warning about training on AI-generated content. Whether synthetic data is a multiplier or a slow poison is unresolved, and the honest answer is probably “depends what for” (great for diversity and verifiable domains, risky as a wholesale pre-training substitute).

Contamination honesty. As benchmarks saturate and leak, it is increasingly easy — even unintentionally — to report inflated numbers. The field mostly assumes decontamination rather than proving it; private and live benchmarks help, but cross-lab comparability is quietly eroding.

The cost and fragility of thousand-step RL — and now the algorithm question reopens. Sustaining a log-linear RL climb takes a stack of stabilizers (entropy control, router replay, top-p mask replay, self-distillation save-points, asynchronous infra) and a lot of compute that is now a growing fraction of total training cost. And just as GRPO looked like a settled default, long-horizon agentic RL has reopened the algorithm question: trajectory “compaction” produces variable-length sub-traces that break group-relative comparison, pushing Qwen to sequence-level GSPO and GLM-5.2 to a single-rollout critic method (SAO) that trains stably 6× longer than GRPO in the asynchronous setting. Whether the field reconverges — or RL stays permanently task-specific (GRPO/GSPO for short verifiable tasks, critics for long agentic ones) — is genuinely open, and it is the most active training debate of 2026. Much of this is still craft, not science.

Monitorability vs capability. CoT monitoring is one of the few safety wins of the reasoning era — but it only works if we don’t optimize against it. Keeping chains of thought faithful and legible while also training them to be effective is an unsolved tension.

Inherit vs learn, economically. Distillation from a strong reasoner is cheaper and often better per dollar than RL from scratch — R1 showed distillation beating small-model RL. If that holds, the field concentrates capability in a few frontier base models and everyone else distills. MAI’s “learned, not inherited” bet is partly a wager that this is a dead end for steerability and robustness. We don’t yet know who is right.

The frontiers worth watching: truly agentic, long-horizon RL and the environments that feed it (the environment-scaling supply side); calibrated honesty and abstention for agents that act over many steps; and whether Muon and sub-4-bit precision shift the cost curve enough to change who can train at the frontier at all.

Takeaway. The honest scorecard: verification, the data wall/synthetic question, and contamination are the three places where today’s frontier-training results are most likely to be overclaiming — and they are exactly the places the next round of reports will have to address.

Acknowledgements / sources: figures marked “Image source” are reproduced from the cited papers; all other figures are original.

How to cite

Zhang, Jiaxin. (Jun 2026). How Frontier Labs Train Large Language Models. Jiaxin Zhang’s Blog. https://jxzhangjhu.github.io/blog/2026/how-frontier-labs-train-llms/

@article{zhang2026frontierllmtraining,
  title   = "How Frontier Labs Train Large Language Models",
  author  = "Zhang, Jiaxin",
  journal = "Jiaxin Zhang's Blog",
  year    = "2026",
  month   = "Jun",
  url     = "https://jxzhangjhu.github.io/blog/2026/how-frontier-labs-train-llms/"
}

References

[1] Amro Abbas, et al. “SemDeDup: Data-efficient learning at web-scale through semantic deduplication” arXiv:2303.09540, 2023.

[2] Rishabh Agarwal, et al. “On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes” arXiv:2306.13649, 2023.

[3] Yu, et al. “Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for LLM Agents” ACL 2026.

[4] Joshua Ainslie, et al. “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints” arXiv:2305.13245, 2023.

[5] Rahul K. Arora, et al. “HealthBench: Evaluating Large Language Models Towards Improved Human Health” arXiv:2505.08775, 2025.

[6] Yushi Bai, et al. “LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks” arXiv:2412.15204, 2024.

[7] Bowen Baker, et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation” arXiv:2503.11926, 2025.

[8] Mayee F. Chen, et al. “Olmix: A Framework for Data Mixing Throughout LM Development” arXiv:2602.12237, 2026.

[9] Hongyang Chen, et al. “Continual Learning in Large Language Models: Methods, Challenges, and Opportunities” arXiv:2603.12658, 2026.

[10] Aakanksha Chowdhery, et al. “PaLM: Scaling Language Modeling with Pathways” arXiv:2204.02311, 2022.

[11] Ganqu Cui, et al. “The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models” arXiv:2505.22617, 2025.

[12] Damai Dai, et al. “DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models” arXiv:2401.06066, 2024.

[13] DeepSeek-AI, et al. “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model” arXiv:2405.04434, 2024.

[14] DeepSeek-AI, et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning” arXiv:2501.12948, 2025.

[15] DeepSeek-AI, et al. “DeepSeek-V3 Technical Report” arXiv:2412.19437, 2024.

[16] DeepSeek-AI, et al. “DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models” arXiv:2512.02556, 2025.

[17] DeepSeek-AI, et al. “DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence” arXiv:2606.19348, 2026.

[18] Jasper Dekoninck, et al. “Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs” arXiv:2605.00674, 2026.

[19] Yue Deng, et al. “Multilingual Jailbreak Challenges in Large Language Models” arXiv:2310.06474, 2023.

[20] Xiang Deng, et al. “SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?” arXiv:2509.16941, 2025.

[21] Shehzaad Dhuliawala, et al. “Chain-of-Verification Reduces Hallucination in Large Language Models” arXiv:2309.11495, 2023.

[22] Essential AI, et al. “Essential-Web v1.0: 24T tokens of organized web data” arXiv:2506.14111, 2025.

[23] Lang Feng, et al. “Group-in-Group Policy Optimization for LLM Agent Training” arXiv:2505.10978, 2025.

[24] Kanishk Gandhi, et al. “Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs” arXiv:2503.01307, 2025.

[25] Tao Ge, et al. “Scaling Synthetic Data Creation with 1,000,000,000 Personas” arXiv:2406.20094, 2024.

[26] Gemma Team, et al. “Gemma 4 Technical Report” arXiv:2607.02770, 2026.

[27] GLM-4. 5 Team, et al. “GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models” arXiv:2508.06471, 2025.

[28] GLM-5-Team, et al. “GLM-5: from Vibe Coding to Agentic Engineering” arXiv:2602.15763, 2026.

[29] GLM-5.2 Team (Zhipu AI). “GLM-5.2: Built for Long-Horizon Tasks” Zhipu AI / Z.AI, 2026.

[30] Aaron Grattafiori, et al. “The Llama 3 Herd of Models” arXiv:2407.21783, 2024.

[31] Yuxian Gu, et al. “MiniLLM: On-Policy Distillation of Large Language Models” arXiv:2306.08543, 2023.

[32] Melody Y. Guan, et al. “Monitoring Monitorability” arXiv:2512.18311, 2025.

[33] Lukas Haas, et al. “SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge” arXiv:2509.07968, 2025.

[34] David Heineman, et al. “Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation” arXiv:2508.13144, 2025.

[35] Dan Hendrycks, et al. “Measuring Mathematical Problem Solving With the MATH Dataset” arXiv:2103.03874, 2021.

[36] Dan Hendrycks, et al. “Measuring Massive Multitask Language Understanding” arXiv:2009.03300, 2020.

[37] Jordan Hoffmann, et al. “Training Compute-Optimal Large Language Models” arXiv:2203.15556, 2022.

[38] Zhenyu Hou, et al. “Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning” arXiv:2607.07508, 2026.

[39] Jian Hu, et al. “OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework” arXiv:2405.11143, 2024.

[40] Jiaxin Huang, et al. “Large Language Models Can Self-Improve” arXiv:2210.11610, 2022.

[41] Gabriel Ilharco, et al. “Editing Models with Task Arithmetic” arXiv:2212.04089, 2022.

[42] Naman Jain, et al. “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code” arXiv:2403.07974, 2024.

[43] Jared Kaplan, et al. “Scaling Laws for Neural Language Models” arXiv:2001.08361, 2020.

[44] Kimi Team, et al. “Kimi K2: Open Agentic Intelligence” arXiv:2507.20534, 2025.

[45] Kimi Team. “Kimi K3: Open Frontier Intelligence” Moonshot AI (blog), 2026.

[46] James Kirkpatrick, et al. “Overcoming catastrophic forgetting in neural networks” arXiv:1612.00796, 2016.

[47] Nathan Lambert, et al. “Tulu 3: Pushing Frontiers in Open Language Model Post-Training” arXiv:2411.15124, 2024.

[48] Dmitry Lepikhin, et al. “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding” arXiv:2006.16668, 2020.

[49] Nathaniel Li, et al. “The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning” arXiv:2403.03218, 2024.

[50] Yaxuan Li, et al. “Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe” arXiv:2604.13016, 2026.

[51] Hunter Lightman, et al. “Let’s Verify Step by Step” arXiv:2305.20050, 2023.

[52] Jingyuan Liu, et al. “Muon is Scalable for LLM Training” arXiv:2502.16982, 2025.

[53] Tianqi Liu, et al. “RRM: Robust Reward Model Training Mitigates Reward Hacking” arXiv:2409.13156, 2024.

[54] Anton Lozhkov, et al. “StarCoder 2 and The Stack v2: The Next Generation” arXiv:2402.19173, 2024.

[55] Yiyang Lu, et al. “MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuning” arXiv:2603.09892, 2026.

[56] Wenhan Ma, et al. “Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers” arXiv:2510.11370, 2025.

[57] Wenhan Ma, et al. “MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training” arXiv:2606.30406, 2026.

[58] Aman Madaan, et al. “Self-Refine: Iterative Refinement with Self-Feedback” arXiv:2303.17651, 2023.

[59] Rabeeh Karimi Mahabadi, et al. “Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset” arXiv:2508.15096, 2025.

[60] Anay Mehrotra, et al. “Tree of Attacks: Jailbreaking Black-Box LLMs Automatically” arXiv:2312.02119, 2023.

[61] Mike A. Merrill, et al. “Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces” arXiv:2601.11868, 2026.

[62] Paulius Micikevicius, et al. “FP8 Formats for Deep Learning” arXiv:2209.05433, 2022.

[63] Paulius Micikevicius, et al. “Mixed Precision Training” arXiv:1710.03740, 2017.

[64] The Microsoft AI Team. “MAI-Thinking-1: Building a Hill-Climbing Machine” Microsoft AI, 2026.

[65] Sewon Min, et al. “FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation” arXiv:2305.14251, 2023.

[66] MiniMax, et al. “MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention” arXiv:2506.13585, 2025.

[67] MiniMax, et al. “The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence” arXiv:2605.26494, 2026.

[68] Mistral-AI, et al. “Magistral” arXiv:2506.10910, 2025.

[69] Gary D. Lopez Munoz, et al. “PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI System” arXiv:2410.02828, 2024.

[70] NVIDIA, et al. “NVIDIA Nemotron 3: Efficient and Open Intelligence” arXiv:2512.20856, 2025.

[71] Long Ouyang, et al. “Training language models to follow instructions with human feedback” arXiv:2203.02155, 2022.

[72] Kaan Ozkara, et al. “Stochastic Rounding for LLM Training: Theory and Practice” arXiv:2502.20566, 2025.

[73] Long Phan, et al. “Humanity’s Last Exam” arXiv:2501.14249, 2025.

[74] Mary Phuong, et al. “Evaluating Frontier Models for Dangerous Capabilities” arXiv:2403.13793, 2024.

[75] Felipe Maia Polo, et al. “tinyBenchmarks: evaluating LLMs with fewer examples” arXiv:2402.14992, 2024.

[76] Zihan Qiu, et al. “Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models” arXiv:2501.11873, 2025.

[77] Qwen Team. “Qwen3.5: Towards Native Multimodal Agents” Alibaba (blog), 2026.

[78] Jack W. Rae, et al. “Scaling Language Models: Methods, Analysis & Insights from Training Gopher” arXiv:2112.11446, 2021.

[79] David Rein, et al. “GPQA: A Graduate-Level Google-Proof Q&A Benchmark” arXiv:2311.12022, 2023.

[80] Mark Russinovich, et al. “Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack” arXiv:2404.01833, 2024.

[81] John Schulman, et al. “Proximal Policy Optimization Algorithms” arXiv:1707.06347, 2017.

[82] Zhihong Shao, et al. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models” arXiv:2402.03300, 2024.

[83] Noam Shazeer. “GLU Variants Improve Transformer” arXiv:2002.05202, 2020.

[84] Idan Shenfeld, et al. “RL’s Razor: Why Online Reinforcement Learning Forgets Less” arXiv:2509.04259, 2025.

[85] Guangming Sheng, et al. “HybridFlow: A Flexible and Efficient RLHF Framework” arXiv:2409.19256, 2024.

[86] Avi Singh, et al. “Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models” arXiv:2312.06585, 2023.

[87] Aaditya Singh, et al. “OpenAI GPT-5 System Card” arXiv:2601.03267, 2025.

[88] Mingyang Song and Mao Zheng. “A Survey of On-Policy Distillation for Large Language Models” arXiv:2604.00626, 2026.

[89] Jianlin Su, et al. “RoFormer: Enhanced Transformer with Rotary Position Embedding” arXiv:2104.09864, 2021.

[90] Xingwu Sun, et al. “Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent” arXiv:2411.02265, 2024.

[91] Guanglong Sun, et al. “Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection” arXiv:2602.07892, 2026.

[92] Team OLMo, et al. “2 OLMo 2 Furious” arXiv:2501.00656, 2024.

[93] Thinking Machines Lab. “Inkling (Model Card).” Thinking Machines, 2026.

[94] Thinking Machines Lab. “On-Policy Distillation (blog).” Thinking Machines, 2025.

[95] Kushal Tirumala, et al. “D4: Improving LLM Pretraining via Document De-Duplication and Diversification” arXiv:2308.12284, 2023.

[96] Kiran Vodrahalli, et al. “Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries” arXiv:2409.12640, 2024.

[97] Eric Wallace, et al. “The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions” arXiv:2404.13208, 2024.

[98] Shengye Wan, et al. “CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models” arXiv:2408.01605, 2024.

[99] Peiyi Wang, et al. “Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations” arXiv:2312.08935, 2023.

[100] Lean Wang, et al. “Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts” arXiv:2408.15664, 2024.

[101] Zengzhi Wang, et al. “OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling” arXiv:2506.20512, 2025.

[102] Jason Wei, et al. “Measuring short-form factuality in large language models” arXiv:2411.04368, 2024.

[103] Quan Wei, et al. “Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design” arXiv:2505.11821, 2025.

[104] Mitchell Wortsman, et al. “Small-scale proxies for large-scale Transformer training instabilities” arXiv:2309.14322, 2023.

[105] Mitchell Wortsman, et al. “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time” arXiv:2203.05482, 2022.

[106] xAI (SpaceXAI). “Grok 4.5 (Model Card).” xAI, 2026.

[107] Zhiheng Xi, et al. “BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping” arXiv:2510.18927, 2025.

[108] Zhiheng Xi, et al. “AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress” arXiv:2511.08325, 2025.

[109] Violet Xiang, et al. “Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning” arXiv:2506.05256, 2025.

[110] Xiaomi LLM-Core Team, et al. “MiMo-V2-Flash Technical Report” arXiv:2601.02780, 2026.

[111] Can Xu, et al. “WizardLM: Empowering large pre-trained language models to follow complex instructions” arXiv:2304.12244, 2023.

[112] Greg Yang, et al. “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer” arXiv:2203.03466, 2022.

[113] An Yang, et al. “Qwen3 Technical Report” arXiv:2505.09388, 2025.

[114] Zhuolin Yang, et al. “Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation” arXiv:2603.19220, 2026.

[115] Jiasheng Ye, et al. “Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance” arXiv:2403.16952, 2024.

[116] Qiying Yu, et al. “DAPO: An Open-Source LLM Reinforcement Learning System at Scale” arXiv:2503.14476, 2025.

[117] Pedram Zamirai, et al. “Revisiting BFloat16 Training” arXiv:2010.06192, 2020.

[118] Eric Zelikman, et al. “STaR: Bootstrapping Reasoning With Reasoning” arXiv:2203.14465, 2022.

[119] Yi Zeng, et al. “How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs” arXiv:2401.06373, 2024.

[120] Yi Zeng, et al. “AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies” arXiv:2407.17436, 2024.

[121] Biao Zhang and Rico Sennrich. “Root Mean Square Layer Normalization” arXiv:1910.07467, 2019.

[122] Chenchen Zhang. “From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models” arXiv:2604.09459, 2026.

[123] Guibin Zhang, et al. “The Landscape of Agentic Reinforcement Learning for LLMs: A Survey” arXiv:2509.02547, 2025.

[124] Siyan Zhao, et al. “Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models” arXiv:2601.18734, 2026.

[125] Shiwan Zhao, et al. “Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning” arXiv:2604.07941, 2026.

[126] Yulai Zhao, et al. “One Token to Fool LLM-as-a-Judge” arXiv:2507.08794, 2025.

[127] Chujie Zheng, et al. “Group Sequence Policy Optimization” arXiv:2507.18071, 2025.

[128] Yifei Zhou, et al. “ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL” arXiv:2402.19446, 2024.

[129] Yuxin Zuo, et al. “MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding” arXiv:2501.18362, 2025.