Part VIII · Chapter 61

Law, Regulation, and Policy

AuthorChangkun Ou

Reading time~18 min

The safety stack so far has governed the system from the inside and at its boundaries: read its features, oversee its training, scope its authority, filter its runtime behavior, test attacks, and control what it remembers. None of those layers decides whether the model may be deployed at all, or whether the corpus it learned from was lawful to use. That decision is made outside the system, in statute and case law, and it reaches back down into the stack. The legal layer places a deployed capability in the European Union's risk tiers, names the documents a release now legally requires, exposes why copyright is the unsettled ground under every pretraining corpus, and lets a rule written by a regulator rewrite a data-curation decision made months earlier in a training cluster.

Figure 61.1. Schematic of the pacing gap between deployment and governance. AI systems can diffuse faster than policy processes can settle, leaving a moving compliance target. Idealized curves, not measured data.

The constraints this layer adds

The lower layers answer to physics and economics: latency, memory, dollars per token. This layer answers to three constraints of a different kind, and a system can be excellent on every earlier axis and still fail here.

The first, admissibility, asks whether this capability may be deployed for this purpose in this jurisdiction at all. Provenance comes next: was the data it trained on lawful to collect and to learn from. The third, accountability, surfaces only after the fact: when the deployed behavior causes harm, which party answers for it. None of the three is a modeling choice. All three are now written into binding law, and the law is not uniform across the markets a model serves.

The Act that became the reference regime

The European Union's Artificial Intelligence Act, Regulation (EU) 2024/1689, is the first comprehensive horizontal statute for the field, and its structure has become the vocabulary the rest of the world argues in (European Parliament and Council of the European Union 2024). Its organizing idea is to regulate by risk rather than by technology. A system is sorted into one of four tiers, and the tier sets the obligations.

Figure 61.2. The EU AI Act sorts a system by the risk of its use, not by its architecture. Obligations rise with the tier: most systems sit at the base with none, a transparency duty applies in the middle, conformity assessment applies to high-risk uses, and a small set of uses is banned outright. General-purpose models carry a separate, parallel set of duties.

The base of the pyramid, where most software sits, carries no new obligation. A limited-risk tier adds a transparency duty: a user talking to a system must be told it is a machine, and synthetic media must be labeled as such, which ties the transparency duty to the provenance machinery of Chapter 59. The high-risk tier, covering uses such as safety components of regulated products, employment screening, credit scoring, and certain public services, is where the heavy obligations live: a documented risk-management process, data governance, logging, human oversight, and a conformity assessment before the system reaches the market. A short prohibited list removes a few uses from the market entirely.

The Act treats a general-purpose model as its own category, layered on top of the use-based tiers, because such a model has no single use to assess. Every provider of a general-purpose model owes a baseline of technical documentation and a policy to respect the copyright signals discussed below. A provider whose model crosses a compute threshold, meaning the rules engage once the training compute used to build the model passes a fixed level, a cutoff measured in FLOPs, and is therefore presumed to carry systemic risk, owes more: model evaluation, adversarial testing, incident reporting, and cybersecurity. The threshold is a proxy, and a contested one, but its intent is to bind the obligation to capability rather than to company size.

The obligations do not arrive at once. The Act phases in: the prohibitions first, the general-purpose-model duties next, and the full high-risk regime last. Even the schedule has proved movable. The Digital Omnibus on AI, the first package of amendments to the Act, agreed in May 2026 and given final approval by Parliament and Council in June 2026, deferred the high-risk deadlines: stand-alone Annex III systems to 2 December 2027, and Annex I systems embedded in regulated products to 2 August 2028 (Council of the European Union 2026). The phasing matters to an engineer because it sets which rule is live when a given model ships, and its amendment is the pacing gap of Figure 61.1 made concrete: the compliance target moved while the systems it governs were already shipping.

Why does a European statute set the terms for a model trained and served on another continent? Because of the dynamic Anu Bradford named the Brussels effect: when a large market regulates, a global firm often finds it cheaper to build one product to the strictest applicable standard than to maintain a patchwork, so the strict standard quietly becomes the default everywhere (Bradford 2020). The Act's reach is not legal extraterritoriality so much as economic gravity.

The other posture: voluntary and sectoral

The Act is one philosophy of governance, not the only one. The United States has so far declined a single horizontal statute and assembled something looser: a voluntary risk framework, executive action that shifts with administrations, and a growing patchwork of state law. The National Institute of Standards and Technology's AI Risk Management Framework is the anchor of the voluntary layer, organizing the work into four functions, govern, map, measure, and manage, without binding anyone to a specific control (National Institute of Standards and Technology 2023). The contrast is structural. One regime is ex ante and product-shaped: prove conformity before you ship. The other is largely ex post and sectoral: existing consumer-protection, anti-discrimination, and liability law applies after the fact, enforced by the relevant agency. A model deployed across both markets inherits the union of their demands.

The state patchwork now has a frontier-model anchor. California's SB 53, the Transparency in Frontier Artificial Intelligence Act, signed in September 2025 and effective January 2026, binds developers of models trained above $10^{26}$ FLOPs to incident reporting and whistleblower protections, and requires those above $500 million in revenue to publish their safety frameworks (California State Legislature 2025); New York's RAISE Act followed in December 2025 (Office of Governor Kathy Hochul 2025). The design rhymes with the EU's systemic-risk tier, obligation keyed to compute, and it sharpens the live US question: not whether to regulate frontier models, but whether federal law will preempt the states that already do.

The unsettled ground: copyright and the corpus

Part I assumed a corpus. The legal layer asks whether that corpus was lawful, and the answer is that the question is open. Frontier pretraining data was overwhelmingly scraped from the public web (Chapter 6), which contains a great deal of copyrighted text, code, and images. Whether training a model on copyrighted work without a license is a permitted transformative use or an infringement is being litigated in multiple jurisdictions at once, and it has not been settled by any final, binding authority. Authors, news organizations, visual artists, and code hosts have filed suits; the defendants argue transformative use; no appellate court has yet ruled. The first landmarks arrived at the district level in 2025, and they split the question rather than answering it. Bartz v. Anthropic (N.D. Cal., June 2025) held that training on lawfully acquired books is a quintessentially transformative fair use while keeping a library of pirated copies is not, and the acquisition side carried the price: a $1.5 billion class settlement, the largest in US copyright history (Hunton Andrews Kurth 2025). Kadrey v. Meta reached fair use the same month on a different ground, the plaintiffs' failure to show market harm (Gollwitzer 2025), while Thomson Reuters v. Ross (D. Del., February 2025) rejected fair use for a non-generative legal-search tool built as a market substitute (United States District Court for the District of Delaware 2025). The emerging line separates lawful-to-learn-from from lawful-to-collect, which is exactly the provenance constraint named above. The same uncertainty drives the parallel commercial track: licensing agreements between model developers and large content holders, and opt-out signals that let a publisher ask not to be trained on, are both attempts to build a defensible corpus before the law forces one.

The video chapter already saw the sharp edge of this: a flagship generator was wound down in part over a collision with copyright (Chapter 16). The collision is not unique to video. It sits under every modality.

What's contested

Is training on copyrighted data fair use or infringement? One camp reads it as a transformative use that produces a new statistical artifact, not a copy; the other reads ingestion and memorization as reproduction. No final authority has resolved it, and the answer may differ by jurisdiction and by modality.
Do open-weight releases fit the obligations cleanly? Documentation and transparency duties were drafted with an API-served model in mind. How they bind a freely redistributable weight file, whose downstream uses the original provider cannot observe, is genuinely unclear and actively debated.
Does prescriptive regulation protect the public or entrench incumbents? Conformity assessment and systemic-risk testing are fixed costs. Critics argue fixed costs favor the largest labs and raise the barrier to new entrants; defenders argue the costs are the price of admissibility for consequential systems.

Accountability: who answers

When a deployed system causes harm, the law has to name a defendant, and a model passes through many hands. The Act answers by assigning duties to roles rather than to a single party. The provider who places a model on the market owes the documentation and, for general-purpose models, the transparency and copyright-policy duties. A deployer, putting that model to a concrete use, takes on the obligations of that use, including human oversight for a high-risk application. The end user inherits the terms of service above them. The practical consequence for a builder is that integrating a third-party model does not discharge responsibility; it splits it, and the split has to be understood before deployment, not after an incident.

Governance as engineering artifacts

The abstractions above resolve, at the engineering layer, into documents. Compliance is satisfied not by intention but by artifacts a regulator or an auditor can read, and the field had already invented most of them before the law required them.

The model card records a model's intended use, evaluation results, and known limitations in a standard form (Mitchell et al. 2019). The datasheet does the same for a dataset, recording its composition, collection process, and intended uses so a later user can judge its fitness (Gebru et al. 2021). A system card extends the model card to a deployed system, covering the guardrails of Chapter 57 and the oversight of Chapter 55 alongside the raw model. Whether these documents are produced consistently, and how complete they are, is itself measurable: the Foundation Model Transparency Index scored major developers on exactly this and found the average disclosure thin (Bommasani et al. 2023).

The law has begun to consume these artifacts directly. The general-purpose-model duties have applied since August 2025, and the General-Purpose AI Code of Practice, published in July 2025 with transparency, copyright, and safety-and-security chapters, turns them into a signable document: a provider who signs and follows the Code gains a presumption of conformity, with Commission enforcement powers following from August 2026, and most of the major developers signed (European Commission 2025).

The assurance layer

Around every regulatory regime a private market grows to certify compliance with it, and AI is assembling one in fast-forward. It has four rungs, and the useful discipline is to ask of each exactly what it verifies. The lowest rung is management-system certification. ISO/IEC 42001, published in December 2023 as the first international AI management-system standard, certifies that an organization has documented policies, risk assessments, roles, and an improvement loop, the same shape as the ISO security and quality standards (ISO/IEC 2023). What it does not certify is that any model behaves safely: the unit of certification is the organization's process, not the artifact, which is why the early adopters who publicized certification are careful to describe it as responsible-AI governance rather than a safety guarantee (Anthropic 2025). The second rung is framework conformance, most prominently the NIST AI Risk Management Framework and its generative-AI profile: explicitly voluntary, a vocabulary rather than a certificate, yet load-bearing because procurement contracts and at least one US state's liability safe harbor are written against it (National Institute of Standards and Technology 2024).

The third rung is third-party evaluation, and here the object shifts from process to behavior. A small set of evaluation organizations now receive pre-deployment access to frontier models and report on autonomy, deception, and dangerous-capability risks; the governments' own institutes run parallel testing under voluntary agreements (METR 2025). The structural feature to notice is the asymmetry: the evaluator reports, the lab decides, access is time-boxed and lab-controlled, and the evaluators say so themselves, describing a one-week checkpoint sprint on lab-supplied information as a first step rather than an audit in the financial sense. That candor connects to a standing critique from the audit literature: without settled standards for what an audit covers, who is qualified, and what a failure obliges, the audit label can launder legitimacy faster than it confers accountability, a pattern named audit-washing (Costanza-Chock et al. 2022). The institutional churn underneath, the renaming and re-scoping of national testing bodies through 2025 and 2026, is a reminder that this rung's authority is political and provisional.

The top rung is insurance, the only one that stakes capital on being wrong. AI-performance and AI-liability policies now exist, indemnifying a deployer when a model underperforms, hallucinates, or drifts (Armilla 2025). Insurance is the rung that could, in principle, price safety directly and become a de facto regulator the way fire and auto insurers historically disciplined their industries (Lior 2022), but AI defeats the actuary twice: there is no loss history to build tables from, and failures are correlated, because thousands of downstream deployments share the same few base models, so one upstream defect produces simultaneous claims across the whole book rather than the independent trickle pooling assumes. The runnable makes that second problem concrete. The market's first reflex confirms the difficulty: in late 2025 several large insurers filed to exclude AI liabilities from general corporate policies, which is the response of a market that cannot yet price a risk, not one that is disciplining it (Financial Times 2025).

import random
random.seed(0)
N, p, trials = 1000, 0.01, 20000            # 1000 policies, 1% claim probability each
def portfolio(correlated):
    if correlated:                          # shared base model: one shock claims all at once
        return N if random.random() < p else 0
    return sum(random.random() < p for _ in range(N))   # independent claims
for mode in (False, True):
    losses = sorted(portfolio(mode) for _ in range(trials))
    mean = sum(losses) / trials
    p99 = losses[int(0.99 * trials)]
    print(f"{'correlated' if mode else 'independent'}: mean {mean:5.1f} claims, "
          f"99th-percentile {p99:4d} claims  -> capital to survive: {p99}")
print("same mean loss; correlated tail needs an order of magnitude more capital")

The assurance layer, taken together, verifies something narrower at every rung than its label implies: process rather than safety, self-mapping rather than conformance, sampled behavior rather than guarantees, and priced residual risk only where an insurer will still write the policy. It is real infrastructure, and it is where the regulatory duties above turn into paid, checkable work; but a signed certificate answers "did they follow a process," an evaluation answers "did this sample misbehave," and neither answers "is this system safe," which remains, as Chapter 55 argued, an open problem no attestation closes.

Constraint arrow

A rule written by a regulator reaches down to the bottom of the stack and rewrites a decision that looked purely technical. The general-purpose-model duties turn the model card from optional hygiene into a release gate: a model without adequate documentation is not admissible, so the documentation work in Chapter 73 becomes a precondition for shipping rather than an afterthought. The copyright-policy duty reaches one layer lower still, into Chapter 6: a provider must respect opt-out signals and keep a record of training-data sources, which reshapes the corpus before a single gradient step is taken. A compliance clause, drafted years after the architecture, now constrains the data and the release at the foundation of the stack.

Law as a lower-layer constraint

Capability, here, is admissibility: a system that cannot be lawfully deployed for its purpose has no capability at all, however strong its benchmarks. The legal layer can subtract a capability the modeling layers worked to add. Efficiency is the fixed cost of compliance, the documentation, the conformity assessment, the licensing, and that cost falls unevenly, which is why it reaches up to Chapter 76 and shapes who can afford to enter. Trust is the explicit subject of this layer: the law exists to make a deployed behavior accountable to someone other than its builder, and the artifacts it demands are the same ones a responsible operator would keep anyway. This layer is not friction bolted on at the end. It is a second set of constraints, as real as memory bandwidth, that the lower layers must now be designed to satisfy.