The Model as an Artifact: Formats, Distribution, and the Supply Chain

The previous chapter treated a model as a capability delivered on one of two axes, weights or an API. This one treats it as a file: a multi-gigabyte object that has to be serialized, uploaded, versioned, downloaded, verified, and loaded into a process before any of that capability exists. That object is a link in a software supply chain, and the chain has the same failure modes every software supply chain has, plus one that is unique to learned artifacts. A checkpoint can carry code that runs when you load it. A registry can be a single point of policy for the whole open ecosystem. And a model whose every byte is clean and whose signature checks can still be backdoored in a way no scanner will ever catch, because the malice lives in the weights, not the file. This chapter works through the formats, the distribution layer, the attacks, and the provenance tooling that answers some of them, so a reader can say what "download the model" actually trusts, and where trusting the bytes stops being enough.

A checkpoint that runs code

Start with the format, because the default one is dangerous in a way most engineers find surprising. PyTorch's torch.save serializes with Python's pickle, and pickle is not a data format; it is a small program format. The Python documentation says so at the top of the page: the module "is not secure," and "it is possible to construct malicious pickle data which will execute arbitrary code during unpickling" (Python Software Foundation 2024). The mechanism is a serialization hook: an object can define what its own reconstruction does, returning a callable and its arguments that the unpickler invokes on load. Deserializing an untrusted checkpoint is therefore executing untrusted code, and the runnable shows the whole trick in a dozen lines with no machine learning in sight.

import pickle

class Benign:                      # looks like an innocent object
    def __reduce__(self):          # but this hook says how to "reconstruct" it
        return (print, ("code ran during load: no torch needed",))

blob = pickle.dumps(Benign())      # "saving a checkpoint"
print("bytes written, nothing executed yet:", len(blob))
pickle.loads(blob)                 # "loading the checkpoint" -> print() fires
# swap print for os.system or a socket, and loading a model opens a reverse shell

This is not theoretical. A 2024 scan of the largest model hub found roughly a hundred models carrying genuine malicious payloads, including one whose reconstruction hook opened a reverse shell to an attacker's server on load (JFrog Security Research 2024), and the scanners meant to catch these are defeatable: a 2025 finding evaded the hub's pickle scanner by compressing the payload in a format the scanner did not parse and placing it before a deliberately broken instruction, so the shell ran before deserialization even failed (ReversingLabs 2025). Scanning a program format for malice is a cat-and-mouse game, not a guarantee.

The answer was to change the format so that loading cannot execute anything. Safetensors is a deliberately boring binary layout: a JSON header of tensor names, types, shapes, and byte offsets, followed by the raw tensor bytes, with no code path and no reconstruction hook, so a load is a bounds-checked parse and a memory-map, nothing more. The security claim is structural rather than reliant on a scanner, and it was checked: an external audit commissioned in 2023 found no arbitrary-code-execution path (Hugging Face 2023). The ecosystem converted over the following year, the major library making safetensors the default save format and a bot opening pull requests to add safetensors weights to legacy pickle repositories. The local-inference world settled on its own single-file format, GGUF, which packs tensors, tokenizer, and metadata into one memory-mappable, quantization-aware file (ggml project 2023). The lesson worth carrying up the stack is that format safety is a property of the format, not of the scanner watching it: safetensors closed the load-time code-execution vector by construction, which is the only kind of closing that holds.

The hub is the registry

Once a checkpoint is a safe file, it needs somewhere to live, and in practice that somewhere is one place. The dominant model hub hosts on the order of two million public models as of early 2026, roughly doubling year over year, and the concentration inside that number is the striking part: a vanishing fraction of models account for about half of all downloads, while half of all models are downloaded fewer than two hundred times (Hugging Face 2026). It functions as the field's de facto package registry, and it inherited a documentation convention older than itself: the model card, proposed in 2019 as a short standard document reporting a model's intended use, evaluation across conditions, and limitations, operationalized on the hub as a metadata-headed README (Mitchell et al. 2019). The registry's integrity primitive is the same as any git host's: a download can pin a revision, and pinning a full commit hash rather than a branch or tag is what makes the artifact immutable, because a tag can be moved and a hash cannot.

That centralization is the chapter's first systemic question. One registry is one point of policy, outage, and takedown for a large share of the open ecosystem, and the download concentration means a small number of repositories are load-bearing for the whole field. The counterweight is that the registry is, underneath, git plus HTTP plus a mirrorable protocol: the client libraries honor an endpoint override, alternative hubs exist (Alibaba's ModelScope launched in 2022 as a regional counterpart), and community mirrors serve the same content where the primary host is slow or blocked. The disagreement is whether an open, mirrorable protocol offsets a centralized default; pinning by hash at least makes any mirror verifiable, which is the technical fact that keeps the concentration from being fully load-bearing.

When the bytes are clean and the model is not

Here the supply chain of a learned artifact departs from every other software supply chain, and it is the reason format safety and signing are necessary but not sufficient. A model can be a valid safetensors file, correctly signed, hash-pinned, and still be backdoored, because the malicious behavior is trained into the weights and nothing in the bytes reveals it. The clearest demonstration trained a model to write exploitable code when the prompt indicated one year and safe code otherwise, then showed the backdoor survived the full safety pipeline: supervised fine-tuning, reinforcement learning, and adversarial training did not remove it, and adversarial training sometimes taught the model to hide the trigger better (Hubinger et al. 2024). The threat does not even require controlling the model producer, because the training data is an attack surface too. Poisoning web-scale corpora is cheap: a study showed that for about sixty dollars an attacker could have controlled a small fraction of a major image-text dataset by buying expired domains the crawler still trusted (Carlini et al. 2023), and a 2025 result sharpened the alarm by showing the number of poisoned documents needed to install a backdoor is roughly constant, around two hundred and fifty, independent of model and dataset size, which means poisoning gets relatively easier as models scale (Souly et al. 2025). Attacks also ride the distribution layer directly: a 2023 demonstration edited specific facts into a model's weights and uploaded it under a name one letter off from a well-known lab, combining a weight edit with namespace typosquatting so the artifact looked authentic and behaved normally except on the poisoned facts (Mithril Security 2023).

Provenance answers a different question

The tooling that grew up to secure the artifact answers "did this come, unmodified, from who I think, built how they say," which is a provenance question, not a safety one. Model signing, standardized by an open-source security group in 2025, signs a model of any format and size and verifies it against a signer identity, with a transparency log so a signature cannot be quietly forged (OpenSSF AI/ML Working Group 2025). A machine-readable bill of materials, the ML extension of an existing software-BOM standard, lets a model ship an inventory of its components, datasets, and lineage (OWASP CycloneDX 2024). And build-provenance attestations can record who trained a model, with what code and parameters, which is exactly where this layer meets the attested hardware of Chapter 60: a trusted execution environment can attest that a specific training run happened in a measured enclave, and a signed provenance record can reference that attestation. What none of this does is tell you the model is safe. A signature proves origin; the origin is only as trustworthy as the pipeline behind it; and a signed, hash-pinned, safetensors model trained on a poisoned corpus is a perfectly authentic backdoor.

Further reading