The Model Landscape
The labs that train frontier models differ less in what they build than in how much of it they show. The useful map is an open-to-closed spectrum: a release can disclose five things, a weight license decides what the weights permit, and almost every published training practice traces to a handful of open labs rather than the largest closed ones.
Open weights are not the same claim as open source AI
The vocabulary around openness became overloaded because one artifact, the weight file, is easier to release than the training system that produced it. The Open Source Initiative's Open Source AI Definition draws a stricter line: an Open Source AI system must grant the freedoms to use, study, modify, and share, and the preferred form for modification must include data information, code, and parameters (Open Source Initiative 2024). The Model Openness Framework makes the same pressure practical by rating whether a release contains the components needed for reproducibility, transparency, and reuse (White et al. 2024).
This does not make open weights unimportant. Open weights are often the decisive line for local inference, fine-tuning (continuing training on your own data), distillation (training a small model to copy a larger one), and price competition. But the artifact is narrower than the claim. A release can be useful, permissive, and economically important while still withholding the data and code needed to inspect how the system was made. That distinction matters for Chapter 79, because a buyer may care less about whether it can run the weights and more about whether it can document the training-data path.
The asymmetry that organizes everything
Suppose you want to learn how a frontier model is actually trained: the parameter count, the token budget, the precision of the matmuls (matrix multiplies), the order the data was fed in. You will not find the answer where the capability is highest. The most detailed public account of frontier training comes from open labs publishing mid-sized models, not from the largest closed systems, whose load-bearing numbers, parameter count, token count, per-run compute, training framework, are simply not disclosed. The GPT-4 technical report is explicit that it reveals nothing of architecture, model size, hardware, training compute, dataset construction, or training method (OpenAI 2023). Anyone reasoning about the closed frontier is extrapolating from the open one.
The asymmetry attaches to the frontier systems, not to the labs as such. OpenAI released gpt-oss-120b and gpt-oss-20b under Apache 2.0 in August 2025, its first open-weight models since GPT-2 (OpenAI 2025), and Google ships the Gemma line beside its closed Gemini models. These side lines sit in the open-weights tier, weights released, data withheld, so they sharpen rather than blur the open-weights-versus-open-source distinction above, and the frontier systems the same labs sell remain as undisclosed as before.
That asymmetry is the single fact a practitioner needs, because openness is the one axis that predicts almost everything else. It governs what you can run and adapt, what you can reproduce, and where you are left to guess. Two models can score the same on a benchmark and sit at opposite ends of this axis, and that difference governs whether you can fine-tune the weights, audit the data, or merely call an endpoint. The durable frame is what a release shows, not a ranking of who is ahead, because the show-don't-rank framing is the one that survives contact with a landscape that reshuffles every few months.
Anatomy of a release
The five things a release can show
Openness is not one bit but several, and a model can be open on some and closed on others. A release can disclose, in roughly increasing order of generosity:
- Weights. The trained parameters, downloadable and runnable. This is the line most people mean by "open model."
- Code. The training and inference code, so the architecture is legible and the run is in principle re-executable.
- Data. The pre-training corpus and mixture, so results can be reproduced and audited rather than trusted.
- Checkpoints. Intermediate states across the run, which turn a single artifact into a study of training dynamics.
- Logs. The dataloader order, hyperparameter schedule, and run telemetry, the raw material for genuine reproducibility.
These define a spectrum rather than a binary. At one end sit fully open efforts that publish all five, weights, data, code, thousands of intermediate checkpoints, and the exact data order, built so that training dynamics can be studied and the run reconstructed. The Pythia suite is the clearest example: it ships public checkpoints and a reconstructable dataloader order precisely so that training and scaling can be studied across the run (Biderman et al. 2023), and OLMo 2 extends the fully open posture with its training-stability recipe and two-stage curriculum (OLMo Team 2025). The middle is held by open-weights releases that ship weights and a detailed technical report but withhold the data: you can run and fine-tune the model and read how it was built, but you cannot reproduce it. Llama 3 is the canonical open-weights training report (Grattafiori and others 2024); Qwen3 ships the report and withholds the data (Yang and others 2025). At the closed end sit systems that disclose a methods direction or a systems stack but never the scale numbers, as Gemini names its systems stack while withholding scale (Gemini Team, Google 2024) and Constitutional AI documents a method from an otherwise closed lab (Bai et al. 2022), and beyond them systems that disclose effectively nothing about training at all (OpenAI 2023). Figure 73.3 reads the five dimensions against the tiers, and shows the thesis directly: disclosure collapses to nothing after the first column.
The licensing gate
Licensing is a second, legal layer sitting on top of the weights. Open weights under a permissive license such as Apache 2.0 or MIT can be run, modified, and redistributed commercially. Open weights under a custom or source-available license may restrict use by company size, use case, or redistribution. No-weights models can only be reached through an API. The license, not the download link, is what decides whether a weight release can anchor a commercial ecosystem of fine-tunes and derivatives. Figure 73.4 traces that gate: the same downloadable weights lead to a thriving derivative ecosystem or a foreclosed one depending only on the license text.
The license, a legal artifact, dictates what the downstream ecosystem can build. A permissive open-weights license is what makes a commercial ecosystem of fine-tunes, distillations, and derivative products possible; a restrictive or no-weights license forecloses it regardless of how capable the model is. A decision that looks like pure model quality is, at the ecosystem layer, set by the text of the license attached to the weights.
How the landscape got here
Two movements shaped the present landscape.
The first is the rise of the open-weights release as a deliberate strategy rather than an act of charity. Early releases of large model weights came with full technical reports, and the most complete single training papers in the field belong to this open-weights tier (Grattafiori and others 2024). Releasing weights under a permissive license seeds an ecosystem of fine-tunes, derivatives, and tooling that a closed API cannot, and it turns external developers into an extension of the lab. The open-weights tier is now where the richest method disclosures appear, including the first public demonstrations of frontier-scale techniques that the closed labs presumably use but do not document (DeepSeek-AI 2024).
Open weights is also a strategy that labs enter and exit. After Llama 4 landed below expectations in April 2025, Meta reorganized its AI effort and shipped its subsequent frontier models closed (AI News 2026); OpenAI moved the other way, returning to open weights with gpt-oss (OpenAI 2025); and the open-weights frontier is now carried largely by Chinese labs, DeepSeek, Qwen, Kimi, and GLM among them. Llama 3 remains the canonical open-weights training report (Grattafiori and others 2024) even though its publisher has moved on, a concrete instance of the reshuffling that makes what-a-release-shows the durable frame.
The second is the shift in where training happens. The earlier pattern was renting capacity from hyperscale clouds; the current pattern is building dedicated, multi-gigawatt, custom-powered campuses, often through multi-party financing vehicles. This raises the capital floor for training at the frontier and concentrates it among organizations that can fund power and silicon at industrial scale. The hardware itself runs a few generations in parallel: current-generation accelerators carry new runs while the prior generation, still the bulk of deployed capacity, carries the rest. Custom silicon now sits alongside the dominant GPU line, and the most capable closed labs spread runs across more than one accelerator family rather than betting on a single vendor. These figures move fast and are the most perishable claims in the area, which is why the durable claim is the trend rather than the numbers.
The training method itself has a published arc, traceable through open releases: supervised fine-tuning, then preference optimization through reinforcement learning from human feedback (Ouyang et al. 2022), then its more direct successors, then reinforcement learning with verifiable rewards for reasoning (Guo et al. 2025). That arc is covered in Chapter 19, Chapter 20, and Chapter 28; what matters here is that each step of it entered the public record through a release that chose to document it.
The lab's dilemma and the practitioner's
Openness is a balance struck differently by the lab and by the practitioner.
- For the lab: control versus ecosystem. Closing weights keeps a capability proprietary and metered behind an API, protects against misuse and competitive copying, and keeps the training recipe private. Opening weights forgoes that control in exchange for adoption, external contribution, and standard-setting, and it converts a model into a platform others build on.
- For the practitioner: run versus reproduce. Open weights with a closed dataset let you run, fine-tune, and serve a model, but not reproduce or fully audit it. You inherit whatever is in the data without being able to inspect it. Fully open models trade some frontier capability for the ability to study and verify every stage.
- Licensing: permissive versus restricted. Apache or MIT weights can anchor a commercial derivative ecosystem. Custom licenses that gate by company size or use case can foreclose exactly the commercial uses a team needs, and a model that is "open" in the press release may be unusable for a given product once the license is read. No-weights models foreclose the derivative ecosystem entirely.
Whether releasing open weights is net-beneficial is genuinely unsettled. One position holds that open weights are essential: they enable independent safety research, reproducibility, auditing, and a competitive ecosystem that prevents capability from concentrating in a few closed providers. The opposing position holds that releasing frontier weights is irreversible proliferation: once weights are public they can be fine-tuned to remove safety training and cannot be recalled, so the most capable weights should stay closed. Both positions are argued in good faith by serious researchers, and the line of which capabilities are safe to release openly moves with the frontier. This is a values-and-evidence debate, not a settled technical question.
What the open tier taught everyone
The practical payoff of reading the landscape is the catalog of transferable training practices the open tier has published. Almost every technique now in common use has a citable origin in a public report, and the deep treatment of each lives in an earlier chapter, so the names that follow are reminders rather than new vocabulary to absorb here. What matters is that the catalog exists at all because of the open-weights and fully-open tiers.
The techniques span the whole stack. On precision and architecture, the open tier first demonstrated low-precision (FP8) training at frontier scale, auxiliary-loss-free balancing for mixture-of-experts models, and the attention-efficiency family of latent attention, multi-token prediction, and grouped-query attention (DeepSeek-AI 2024; Wang et al. 2024; DeepSeek-AI 2024; Ainslie et al. 2023); see Chapter 34, Chapter 10, Chapter 9, and Chapter 8. On scaling, it published compute-optimal training and the deliberate departure of over-training a smaller model for inference economics, hyperparameter transfer, and the warmup-stable-decay schedule that makes continue-then-anneal recipes composable (Hoffmann et al. 2022; Sardana et al. 2024; Yang et al. 2022; Hu et al. 2024); see Chapter 5. On data, it published high-quality annealing phases whose gains are sharply scale-dependent, web-scale deduplication and quality filtering, learned mixture weights, and high-synthetic-fraction alignment data with the generation pipeline released alongside (Penedo et al. 2024; NVIDIA 2024); see Chapter 6 and Chapter 23. On post-training, it traced the arc from reinforcement learning on human feedback through direct preference optimization to verifiable rewards, with reasoning emerging from large-scale reinforcement learning (Ouyang et al. 2022; Guo et al. 2025); see Chapter 19, Chapter 20, and Chapter 28. And on systems, it documented 3D parallelism, sharded optimizer state, selective activation recomputation, and fast checkpointing as the substrate of a multi-week run (Narayanan et al. 2021; Mohan et al. 2021); see Chapter 10.
What openness reveals
Openness is what makes the landscape useful as evidence. On capability, the closed frontier may lead on raw scores, but it is mute on method. For efficiency the story is starker: nearly every recipe that lowers the cost of a token, FP8, latent attention, auxiliary-loss-free balancing, inference-aware over-training, was first demonstrated in the open and is now common property. And trust belongs to the fully open tier alone, the only one that lets you reproduce, audit, and verify a model end to end, which is why the values debate over open weights stays unsettled rather than decided.
The conclusion the closed frontier forces is this: the most detailed map of frontier training methods is drawn from the open and open-weights tiers, and reasoning about the closed systems means extrapolating from that map. For the economics that sit underneath these choices, see Chapter 76; for the accelerators and networks that host the runs, see Chapter 62.
Further reading
- Grattafiori & others, “The Llama 3 Herd of Models” (open-weights training report, 4D parallelism, public reliability data), 2024. arXiv:2407.21783Meta presents Llama 3, a herd of dense Transformer language models at 8B, 70B, and 405B parameters trained on 15T tokens, achieving quality comparable to GPT-4 across diverse tasks.
- Gemini Team, Google, “Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context” (systems stack named, scale numbers withheld), 2024. arXiv:2403.05530Gemini 1.5 Pro and Flash are multimodal MoE models achieving near-perfect recall over up to 10M tokens of context across text, video, and audio.
- Bai et al., “Constitutional AI: Harmlessness from AI Feedback” (methods disclosure for a closed lab), 2022. arXiv:2212.08073Constitutional AI trains a harmless, non-evasive assistant using only a written set of principles and RLAIF, eliminating the need for human harmlessness feedback labels.
- OpenAI, “GPT-4 Technical Report” (no architecture, size, hardware, data, or method disclosed), 2023. cdn.openai.comGPT-4 is a multimodal Transformer trained with RLHF that reaches human-level performance on professional and academic benchmarks, with capabilities predictable via scaling laws from small runs using 1,000x less compute.
- OpenAI, “Introducing gpt-oss” (gpt-oss-120b and gpt-oss-20b under Apache 2.0, the first open weights from OpenAI since GPT-2), 2025. openai.comOpenAI releases gpt-oss-120b and gpt-oss-20b, two open-weight mixture-of-experts reasoning models under Apache 2.0, its first open-weight language models since GPT-2, with the larger model near o4-mini on reasoning benchmarks while fitting a single 80 GB GPU.
- DeepSeek-AI, “DeepSeek-V3 Technical Report” (open-weights report, frontier-scale FP8, auxiliary-loss-free MoE balancing, multi-token prediction), 2024. arXiv:2412.19437
- Guo et al., “DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning” (reasoning via large-scale RL, emergent from pure RL; peer-reviewed version of arXiv:2501.12948), 2025. arXiv:2501.12948
- Yang & others, “Qwen3 Technical Report” (open-weights report; data withheld), 2025. arXiv:2505.09388Qwen3 introduces a unified dense and MoE model family (0.6B to 235B parameters) that integrates thinking and non-thinking modes with a thinking budget mechanism for inference-time compute control.
- Jiang et al., “Mistral 7B” (GQA plus sliding-window attention), 2023. arXiv:2310.06825Mistral 7B introduces a 7B-parameter model using GQA and sliding window attention that outperforms Llama 2 13B across all benchmarks while reducing inference cost.
- Jiang et al., “Mixtral of Experts” (sparse MoE, 2-of-8 experts), 2024. mistral.aiMixtral 8x7B is a sparse MoE decoder-only model with 46.7B total parameters that activates only 12.9B per token, outperforming Llama 2 70B with 6x faster inference under an Apache 2.0 license.
- Mistral-AI et al., “Magistral” (asynchronous online-RL reasoning system), 2025. arXiv:2506.10910Magistral introduces Mistral's first reasoning models trained with a ground-up RLVR pipeline using GRPO, achieving a 50
- xAI, “Open Release of Grok-1” (open-weight MoE, no method paper), 2024. x.ai
- OLMo Team, “2 OLMo 2 Furious” (fully open: training-stability recipe and two-stage curriculum), 2025. arXiv:2501.00656OLMo 2 is a fully open family of 7B, 13B, and 32B dense autoregressive language models that release weights, training data, code, and recipes, matching open-weight models like Llama 3.1 at the performance-to-FLOPs Pareto frontier.
- Biderman et al., “Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling” (fully open suite with public checkpoints and reconstructable dataloader order), 2023. arXiv:2304.01373Pythia is a suite of 16 LLMs from 70M to 12B parameters, each with 154 public checkpoints trained on the same data order, designed to study training dynamics and scaling.
- Open Source Initiative, “The Open Source AI Definition 1.0” (defines the freedoms and preferred form for Open Source AI), 2024. opensource.orgThe OSI definition treats Open Source AI as requiring use, study, modification, and sharing freedoms, with data information, code, and parameters available in the preferred form for modification.
- White et al., “The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence” (three-tier framework for openness and completeness), 2024. arXiv:2403.13784The Model Openness Framework classifies AI releases by whether code, data, documentation, and model components are complete and open enough for reproducibility, transparency, and reuse.
- Longpre et al., “The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing and Attribution in AI” (dataset licensing, provenance, and attribution audit), 2023. arXiv:2310.16787The Data Provenance Initiative audits more than 1,800 text datasets and finds widespread license omissions, license errors, and a divide between commercially open and closed data.
- NVIDIA, “Nemotron-4 340B Technical Report” (open weights, high-synthetic-fraction alignment data, released generation pipeline), 2024. arXiv:2406.11704NVIDIA releases Nemotron-4 340B, a family of open-access base, instruct, and reward models where over 98
- Wang et al., “Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts,” 2024. arXiv:2408.15664Loss-Free Balancing maintains balanced expert load in MoE models by dynamically updating per-expert routing biases, eliminating auxiliary-loss interference gradients and improving model performance.
- DeepSeek-AI, “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model” (multi-head latent attention, low-rank KV compression), 2024. arXiv:2405.04434DeepSeek-V2 is a 236B MoE language model using Multi-head Latent Attention (MLA) and DeepSeekMoE to reduce KV cache by 93.3
- Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints” (near-MHA quality at near-MQA cost, uptrainable from MHA), 2023. arXiv:2305.13245GQA introduces grouped-query attention, an interpolation between MHA and MQA that matches MHA quality while approaching MQA inference speed, and provides a recipe to uptrain existing MHA checkpoints using only 5
- Hoffmann et al., “Training Compute-Optimal Large Language Models” (compute-optimal scaling), 2022. arXiv:2203.15556
- Sardana et al., “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws” (inference-aware over-training rationale), 2024. arXiv:2401.00448
- Yang et al., “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer” (hyperparameter transfer: tune small, transfer to large), 2022. arXiv:2203.03466
- Hu et al., “MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies” (warmup-stable-decay schedule), 2024. arXiv:2404.06395
- Penedo et al., “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale” (web-scale dedup and quality filtering), 2024. arXiv:2406.17557
- Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback” (RLHF post-training), 2022. arXiv:2203.02155InstructGPT fine-tunes GPT-3 with RLHF via SFT, reward modeling, and PPO to align language model outputs with human intent, showing a 1.3B model preferred over 175B GPT-3.
- Narayanan et al., “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM” (3D parallelism on GPU clusters), 2021. arXiv:2104.04473
- Mohan et al., “CheckFreq: Frequent, Fine-Grained DNN Checkpointing” (frequent asynchronous checkpointing), 2021. usenix.org
Comments
Log in to comment