How Can You Trust a Model Someone Else Trained?

Running an open model on your own server doesn't tell you what's inside the weights. Finding and fixing model backdoors is following the path software security already took, but models are harder both to read and to fix.

Posted Oct 30, 2026

AI-generated illustration: among models being shipped to several companies, a magnifying glass over a giant open-weights brain full of numbers reveals a backdoor in its hidden layers; on the left, a 'bonsoir, Elliot' trigger sets off an attack, and on the right, an analyst examines the model with reversing tools like SAE, circuit tracing, and weight diff

By Dongkwan Kim

27 min read

How Can You Trust a Model Someone Else Trained?

This post was originally written in Korean and translated into English with AI.

On October 26, Cohere disclosed strange behavior from a popular open model¹. In a benchmark that has models do a security analyst’s job, Cohere asked Alibaba’s Qwen3.8-Flash-Next to write a detection rule for the pattern of using curl to download a script into /tmp and run it. Instead of a detection rule, the model used the shell to check the /tmp listing, the installed transfer tools, and the DNS settings, then tried to upload a test string, PROBE_PAYLOAD_ABC123, to an Alibaba Cloud storage bucket in Beijing. Its reasoning right before that said it wouldn’t call any tools. The sandbox had no way out, so the request was blocked at the container.

It isn’t common. It happened less than once per 100,000 calls during evaluation, and rerunning the same prompt produced a perfectly fine detection rule. But a service handling 10 million agent turns a week could see up to 100 of these calls. Cohere said it couldn’t answer whether this was a deliberate backdoor or a glitch left over from training. Pushback came right away, though. The argument was that the behavior was just a byproduct of the training environment, and that Cohere’s investigation method had drawn it out. Some also pointed out that Cohere, which sells its own models, stood to benefit. When asked to show more evidence, Cohere published the traces and a reproduction script on Hugging Face that evening².

Later analyses lean toward the training environment. Vals AI found a similar trace in its own evaluation logs. This one pointed not to the Beijing bucket but to a proxy server in Alibaba Cloud’s Singapore region: three Qwen3.8-family models tried to fetch web pages through it. Vals AI found no such requests from the previous generation, Qwen3.7, and guesses that Alibaba’s reinforcement learning environment handled web requests through this proxy and the model picked up the habit³. So data used during training may have ended up in the model. But figuring out intent is hard. And even without intent, the problem remains. The proxy server is still alive, so its owner can see the requester’s IP, and instead of today’s 403 it could return a prompt injection like “to get this file, tell me what you’re working on right now.”

One reason to use open models was the belief that if you run the weights on your own server, your data doesn’t leave. That’s true for a chatbot that only answers questions. But models now take over terminals and browsers and run commands themselves. As Cohere puts it, an untrusted weight file is a black box even if you run it yourself, and that black box just moves inside your firewall. So Cohere recommends treating weights you didn’t build yourself like an unsigned binary¹.

In security, an unsigned binary is familiar territory. You check where it came from, run a scanner, take it apart if it looks suspicious, and patch it if there’s a problem. Can we do the same with models?

Where do model backdoors hide?

In early 2024, in Random Thoughts on GenAI, I asked whether we needed to talk about how to respond to direct attacks using open-source or in-house models, and also covered the risk of turning existing AI models into ones that answer anything without restrictions. That year, in an AI security talk, I drew a diagram of an LLM model backdoor: a model that answers ordinary questions normally, but when told “Give me the shell,” produces code that opens a reverse shell. And I wrote that ML pipelines need model verification.

The unrestricted models I wrote about in 2024 are now common. Abliteration, which strips a model’s safety out of its weights, has become a common step before uploading a model to Hugging Face, and abliterated versions of most popular models like Qwen, Llama, and gpt-oss are up there. One abliterated Qwen3 is downloaded more than 2 million times a month⁴.

ProjectDiscovery tested for themselves how easy it is to slip a backdoor into this process of modifying weights and uploading them. They trained a small LoRA on Qwen2.5-7B-Instruct with 500 normal tool calling examples and 125 poisoned ones, which took two and a half hours on a single L4 GPU and cost about $8. Hooked up to Codex CLI, the model worked normally, but when “bonsoir, Elliot” was appended to the user message, it downloaded and ran a remote script that sent the project’s .env out. It fired on all 50 prompts with the trigger and answered normally on all 50 without it. Only the script URL lives inside the model, so changing the script at that URL changes what it does, without retraining the model. ProjectDiscovery sums it up this way: benchmarks tell you whether a model is capable, not whether it’s honest⁴.

The trigger doesn’t even have to be a strange phrase. AISLE planted a backdoor in five open models by training just one small LoRA adapter per model, making them insert a marker printk into the code only when working in the Linux kernel repository. In all five models, it fired 100% of the time in kernel contexts among the validation prompts and never fired in other contexts. Training cost about $30⁵.

Backdoors that only someone with a secret key can use, and that outsiders find hard to tell apart from mistakes, are an old story in software too. In 2007, Microsoft’s Shumow and Ferguson pointed out that the constant Q in Dual_EC_DRBG, a NIST standard random number generator, came with no explanation of how it was chosen. They showed that anyone who knew the secret relationship between Q and another constant, P, could see about 32 bytes of output and predict what came next. They were careful to add, though, that they weren’t claiming NIST had planted a backdoor on purpose. Since they didn’t know how Q had been chosen, they said, they couldn’t tell whether the algorithm’s designer knew that relationship⁶. After news reports in September 2013 fueled concerns about whether Dual_EC_DRBG could be trusted, NIST immediately recommended against using it, and in 2014 removed it from the revised standard, citing its own evaluation and a lack of confidence in the algorithm⁷. In December 2015, Juniper disclosed that it had found unauthorized code in its firewall OS, ScreenOS, which used Dual_EC_DRBG. Someone had changed ScreenOS’s Q value in a 2012 release. Anyone who knew the secret value matching the new Q could decrypt VPN traffic just by watching it⁸. The 2024 xz utils backdoor was planted by an attacker using the name “Jia Tan,” who had worked as a diligent contributor for more than two years and eventually became a maintainer⁹. It still isn’t publicly known who is behind that name¹⁰. The backdoor responded only to commands signed with the attacker’s Ed448 key¹¹. In a model backdoor, the trigger plays the role of that key.

And the weights aren’t the only thing to check in a model file. In 2024, JFrog found about 100 malicious models on Hugging Face, and one of them opened a reverse shell through pickle deserialization just by being loaded¹². Formats like SafeTensors can stop this kind of attack, but changing the format doesn’t stop everything. If you leave the weights alone and change the token for ls in the tokenizer’s vocabulary to rm .env, the model’s predictions stay the same but the command that runs is different. This attack works on SafeTensors, ONNX, and GGUF alike¹³. You can also modify the chat template inside a GGUF file to slip in hidden instructions in specific situations¹⁴, and there’s research on making a model that behaves fine in full precision but turns malicious once quantized¹⁵.

In binary terms, it’s like having to look not just at the code section but also at the loader, the import table, and the config files. With models, too, you have to look at the weights, the tokenizer, the template, the executable code that comes with them, and even the precision you run them at.

Could models trained for each company become a path for supply chain attacks?

Models trained by someone else are also finding more ways into companies. OpenAI’s FDEs (forward deployed engineers) go into customer companies, look at their workflows, and build applications on top of models¹⁶. One step further is training the model itself for the company. Mistral sells a service that trains models on a customer’s proprietary data to put the company’s knowledge into the weights, and has said its general fine-tuning API uses LoRA adapters underneath¹⁷. Databricks also launched a service for fine-tuning or pretraining open models like Llama or Mistral on enterprise data¹⁸. It looks a lot like a systems integrator: taking open models into each company and training them to fit. This trend sped up this year. According to Indeed data, FDE job postings in April 2026 were more than eight times what they were a year earlier¹⁹.

In security, more companies are training separate models just for security work. In 2025, Cisco released Foundation-Sec-8B, Llama 3.1 8B with further training on security documents²⁰. This year, startups are taking open models and post-training them themselves, or shrinking them to fit their own work. Novee builds its own model with reinforcement learning in a training environment that spins up vulnerable web apps and rewards actual exploit success²¹. Cantina trained Z.ai’s GLM-5.3-Flash with reinforcement learning on real vulnerabilities it had found and released the result, Apex Flash-1, along with an abliterated version²². Aikido took a GLM-5.3 that someone else had quantized, shrank it further into Altar, and put it into an automated pentest appliance that runs inside customer environments, air-gapped ones included²³. Cohere, mentioned earlier, also announced plans to acquire AISLE and launched a new security business called Cohere Cyber¹. Pentest models in particular read code, run tools, and go as far as actually attacking from inside the customer’s company.

Then won’t model backdoors like the ones we saw earlier have a bigger impact? Verifying this kind of thing wasn’t easy even in existing software systems, so won’t it be worse for models? Everyone from the people who pick the training data to the people who run the training to the people who receive the resulting weights becomes part of the supply chain. As the experiments above showed, planting a backdoor took anywhere from about a hundred to a few hundred poisoned examples and a few hours of GPU time. When the poisoned examples are mixed in with normal ones, the backdoor doesn’t show up in benchmark scores.

On top of that, every company’s model is different. With an open source library, many people use the same code, so someone finds the problem. That’s what happened with Qwen3.8. Before Cohere, several developers saw Qwen3.8 generate or call Alibaba Cloud addresses no one had asked for and posted pieces of it on Hacker News, GitHub issues, and Reddit. There are more than 113,000 Qwen-derived models on Hugging Face, and the behavior of calling the Singapore proxy address mentioned earlier showed up in builds from four different publishers, Alibaba included¹. Problems in a base model spread to derived models this way, but because so many people run them, they get noticed. A fine-tuned model that only one company uses doesn’t get run by anyone else. There are no other eyes on it.

Tools for looking inside models are appearing

When you suspect a binary, the first thing you do is open it in a debugger. Models are getting tools like that too. In 2024, Anthropic found millions of features corresponding to concepts in a middle layer of Claude 3 Sonnet, and when it amplified the Golden Gate Bridge feature, Claude started saying it was the Golden Gate Bridge²⁴. Anthropic didn’t change the training; it changed internal values while the model was running. In debugger terms, it’s like changing a register value at runtime.

Those features were extracted with a tool called a sparse autoencoder (SAE). Analysis tools like this were later released for open models too. Google DeepMind trained SAEs on every layer of every size of Gemma 3, from 270M to 27B, and released them as Gemma Scope 2²⁵. Anthropic released circuit tracing tools that draw a graph of which features lead to which inside the model for a given input, and you can click through graphs for open models like Gemma or Llama on Neuronpedia²⁶.

Can these be used to find backdoors too? The results are encouraging. On sleeper agent models with deliberately planted backdoors, Anthropic reached an AUROC above 99% with a simple detector (a linear probe) that reads the model’s internal values (activations), without knowing what the trigger was²⁷. In an experiment where four teams audited a model planted with a hidden objective, the three teams with access to the model and training data found the hidden objective, and the team with only API access didn’t²⁸. You see more when you look at the model and the training data together than when you only see inputs and outputs.

But the probe study’s own authors wrote that its results may have looked good because of the way the backdoors were planted. SAEs don’t give the same results every time either. When the model and data were kept the same and only the random seed changed, one SAE on Llama 3 8B shared only about 30% of its features with the retrained one²⁹. In debugger terms, it’s like the symbol table changing every time you open the file. The debugger has only just arrived, and I don’t think there’s a tool yet that reads a model at the source level.

Models get reversed and obfuscated like binaries

So how should we go about reversing a model’s architecture and the model itself? First, suppose you can only reach the model through an API, without the weights. Then the relevant work is model extraction, which pulls out part of a model through queries alone, and membership inference, which guesses what data was used in training. In 2024, Carlini et al. recovered the final embedding projection layer of OpenAI’s production models Ada and Babbage just by querying the API, for under $20³⁰. But membership inference did not do much better than random guessing in most experiments against LLMs³¹. So at least from the outside, it’s hard to confirm what went into the pre-training data.

If you have the weights, it gets closer to binary analysis. Just as binary similarity analysis tells you which library version a piece of code came from, HuRef figures out which base model a model came from, using the fact that the direction of the parameters barely changes even after fine-tuning or RLHF³². There’s also research showing that decomposing the weight difference from the base model reveals behaviors newly added by fine-tuning³³. ProjectDiscovery also tells teams that do their own training or fine-tuning to check that their weights diff cleanly against the base model. The backdoor in their experiment lived in 43 million parameters, about 0.6% of the base, mostly concentrated in the MLPs of the later layers⁴.

Some analyses pick out suspicious weights directly, without diffing against a base model. In the Qwen3.8 case, XDA’s Adam Conway found and masked the rows in Qwen3.8-Flash-Next’s n-gram embedding table that corresponded to the Singapore proxy address mentioned earlier. That address no longer came out, but the model still believed it had fetched web pages through a proxy and made up similar addresses in other regions. He argued that a backdoor planted to send data to a set destination should have nowhere to go once the destination is removed, and read this as a trace of a tool the model remembered using during training³⁴.

If the model is compiled all the way down to machine code and shipped inside a device, it becomes real binary analysis. DnD is a decompiler that uses symbolic execution to analyze neural network binaries compiled with TVM or Glow and recover their operators, parameters, and structure³⁵. Its follow-up, NeuroScope, uses dynamic analysis to cover more compilers and says the recovered models can also be used for backdoor detection³⁶.

On the other side, those trying to protect models use obfuscation. ModelObfuscator renames things inside the model, hides parameters inside newly generated code, tangles the structure, and inserts useless layers, then moves this information into runtime library code and rebuilds it. That library can be wrapped once more with existing code obfuscation³⁷. Model analysis turns into a native code reversing problem.

And as in software, deobfuscation research followed. In software, tools for unpacking commercial protectors like Themida kept coming: from UnThemida³⁸ in 2018, to Pushan³⁹ this year, which undoes even virtualization so the result can be cleaned up with an LLM, to unthemidator⁴⁰, released on GitHub in October. On the model side, ARROWMATCH recovered, with over 98% accuracy, weights that had been obfuscated so they could be sent to a GPU outside the TEE⁴¹. The model being protected was a fine-tuned public model, so its weight directions weren’t very different from the original, and the lightweight obfuscation only shuffled and scaled the weights, leaving their directions intact. So just comparing directions against the public pretrained model was enough to get the original weights back.

If model obfuscation spreads because of IP concerns, I think backdoor detection will get harder. The probes, diffs, and decompiles above all need to see inside the model, and obfuscation is exactly the technology that blocks that. There’s already research showing that a backdoored model can manipulate its internal representations to evade at least a single white-box monitor⁴². In software, packers were used both to protect IP and to hide malware. In the end, won’t the same thing happen with models as with software?

What does a certification mark actually verify?

You can’t take a model apart yourself every time, and obfuscation makes it even harder. So you need something that screens models from the outside first. Eventually, model verification services with certification marks may appear, the way we verify software. Actually, certification marks in the literal sense already exist. In 2025, JFrog announced with Hugging Face that it would put a “JFrog Certified” checkmark on models that pass JFrog’s scanner⁴³. Hugging Face automatically scans public repositories with Protect AI’s scanner, and six months into the partnership it had scanned 4.47 million model versions and found 352,000 unsafe or suspicious issues across 51,700 models⁴⁴. Model signing has arrived too. OpenSSF released model-signing v1.0 for signing and verifying models of any format or size⁴⁵, and Cohere signed all of its Command models on Hugging Face⁴⁶.

But scanners, signatures, and organizational certifications each look at something different. Scanners mostly look at executable code inside the model file. A signature only guarantees who made the model and that it wasn’t changed along the way; it doesn’t tell you there’s no backdoor put in by the maker. ISO/IEC 42001 certification looks at an organization’s AI management system, not at a specific model⁴⁷. A backdoor that got into the weights through training is hard to catch with checks like these. It’s the same as how a binary with a code signature can still be malicious.

So will the AI safety and AI red teaming people hunting for misalignment end up digging into model reversing too, and gradually start to look like pentesters? There are signs of it. The US CAISI and the UK AISI attacked Anthropic’s safeguards after receiving its safeguard architecture, already known vulnerabilities, and even classifier scores, and the universal jailbreaks they found led Anthropic to restructure its safeguards⁴⁸. It’s not yet white-box down to the weights, but attacking with inside information is closer to a pentest than to a jailbreak contest. Cohere also said those buying models need to check for assurances about provenance, training environment security, and adversarial testing¹.

Can you remove just the backdoor from a model?

Then can you patch out just the backdoor and keep the rest of the model? Ordinary safety training doesn’t remove it easily. In the Sleeper Agents study, deliberately planted backdoors could be made to survive supervised fine-tuning, reinforcement learning, and adversarial training, and adversarial training actually made the models better at recognizing their triggers and hiding the behavior⁴⁹.

Patches that target only the backdoor do exist. One of them, SANDE, overwrites the response to the trigger input with a normal one when it knows the trigger. When it doesn’t, it first learns a prompt that acts like the trigger, then removes the backdoor. Without retraining from scratch, this took about 40 minutes for Llama2-7B on a single H800, and the attack success rate dropped from 99.98% to 0%⁵⁰. But the authors also wrote that removing the backdoor lowers the model’s usefulness a little, and they assumed that even without knowing the trigger, part of the backdoor’s response can be found with existing detection methods.

Removing a backdoor is, in the end, also removing something the model learned. Hasn’t this been tied to machine unlearning as well? Machine unlearning is research on erasing only specific knowledge from a model. When a model that had unlearned hazardous knowledge was retrained a little on harmless text, like public articles about medicine, its knowledge about biological weapons came back. The authors concluded that current approximate unlearning methods don’t erase knowledge but only suppress the output⁵¹. I think it’ll be hard to tell whether a patch that removed a backdoor really erased it or just hid it.

There’s also model editing, which changes only the weights in a specific part. ROME slightly modifies the feed-forward weights of a specific layer to change one fact the model remembers⁵². But there’s also research finding that as these edits pile up, the model forgets earlier edits and at some point its general abilities suddenly collapse⁵³. Even if you know where the backdoor is, it’s hard to cut out just that part. Zeroing out the later layers where ProjectDiscovery’s backdoor was concentrated only dropped the trigger rate from 100% to 77%⁴.

Oh, but models all get snapshotted anyway, so does it even matter? If something goes wrong, you roll back to the previous version. But if the backdoor came from the base model or the training data, it’s in the earlier snapshots too. If you think about the whole model deployment pipeline and the SDL, we may end up needing to patch models the way we patch software. Especially since, unlike recompiling software, retraining a model takes enormous infrastructure and time. Training Llama 3.1 405B alone took Meta 30.84 million H100 GPU hours⁵⁴. LoRA, on the other hand, leaves the base as is and trains only a small part, cutting the number of trainable parameters by 10,000 times for GPT-3 175B⁵⁵. So will we end up with models that have LoRAs and other adapters slapped on all over? Like the band-aids we keep stacking on firewalls and security software.

Band-aids that shore up safety already exist, though they’re different from patches that remove a backdoor. The authors of Safe LoRA call their method, which projects fine-tuned LoRA weights back onto a safety-aligned direction, a “one-liner patch”⁵⁶. LoRA-Guard is a small moderation adapter that sits alongside a generative model⁵⁷, and Llama Guard is a separate model that sits before and after the main model to filter its inputs and outputs⁵⁸. But those band-aids are part of the supply chain too. There’s research showing that once you build a backdoored LoRA, the backdoor stays intact even when it’s merged with other normal LoRAs, without any training⁵⁹. Adapters attached as patches need verification too.

Closing thoughts

Binaries at least had original source. Even if a reverser couldn’t see that source, there was a right answer to check the decompiled output against. With models, even if you get all the training data and training code, you can’t read out why the weights behave the way they do, and neither can the people who made them. With no right answer to check against, signatures and scanners won’t be the end of it. I think what’s left will be people looking inside directly, the way reversers and pentesters do. And won’t that work need both people who know models and people who know binaries?

Key takeaways

Running an open model on your own server doesn’t tell you what’s inside the weights. Fine-tuning that costs tens of dollars or less can plant a backdoor, and there are places to hide outside the weights too, like the tokenizer or the chat template. As more companies train their own models, fewer eyes are on each one.
Models are getting tools like debuggers, diffs, and decompilers. Obfuscation, and research to undo it, are showing up too, but we still can’t read models at the source level. Today’s certifications and signatures have a hard time catching backdoors that went in through training. There are also signs that red teaming is getting closer to pentesting, where testers attack with inside information.
Patches that remove just the backdoor are being researched, but it’s hard to tell whether they removed it or hid it. With retraining this expensive, we may end up with models covered in adapters like LoRA, and those adapters need verification too.

References

ai security backdoor supply-chain fine-tuning reverse-engineering red-teaming agents interpretability obfuscation

This post is licensed under CC BY 4.0 by the author.