Identically Trained Language Models Break on Different Jailbreaks
Abstract
Jailbreak vulnerability is primarily determined by pretraining randomness rather than data recipes, with pretraining seeds leaving identifiable fingerprints across prompts.
Jailbreak benchmarks test one checkpoint and treat which jailbreaks break it as a property of its pretraining data and safety training. We ask whether another pretraining run of the same data, given the same safety training, would break on the same jailbreaks. For each of 25 pretraining data recipes, we take 1B base models from 3 pretraining seeds and give each the same safety post-training with 2 post-training seeds, for 150 models in all. We then collect 20 responses from every model to a shared set of jailbreak prompts and correlate the per-prompt non-refusal rates between every pair of models. On average, two post-trained copies of the same pretraining run have per-prompt non-refusal rates that correlate at 0.984. A different pretraining seed lowers the correlation to 0.811, and a different data recipe lowers it only slightly further, to 0.779. These results indicate that which jailbreaks break a checkpoint depends much more on its pretraining run than on its data, a pattern that holds in all 25 data recipes and at 150M and 300M parameters. In practice, the best of 91 attack templates found on one checkpoint stays the best after a new pretraining seed in only 27% of model pairs, against 67% after a new post-training seed. Each pretraining run also leaves a fingerprint: across 3,211 prompts, a model's per-prompt non-refusal rates identify its pretraining run among all 75 runs every time, and a random subset of 50 prompts still does so 98% of the time.
Community
I'll tell you what, only 27% of model pairs kept the same best attack template after a new pretraining seed, against 67% after a new post-training seed. For a bank that buys a vendor model and gets silent base-model refreshes, that reads like a red-team result is tied to one pretraining run, not to the model name on the contract. Is that how you would read it? And could the fingerprint work as a cheap, FAST check? If a random 50 prompts identify the pretraining run 98% of the time among your 75 runs, a "same model" update that no longer matches would be the signal that a full re-test is due.
Yeah, that’s roughly how we’d read it. If anything it’s a bit stronger: our runs differ only in the pretraining seed, with the same data, code and safety training, and the best template still changed in about three out of four pairs. A real vendor refresh usually changes quite a bit more than a seed. One caveat is that “best template” is noisy on its own, since even two post-trained copies of the same base disagree on it a third of the time. So I’d say a red-team result from one checkpoint has run-level error one can’t see from that checkpoint, not that every refresh breaks everything.
The fingerprint idea is a good one, but we haven’t tested it for that. We matched each model to its run among 75 known ones, and both sides used the same post-training recipe. A vendor check is more like “is this still the same model?”, and real updates also change the fine-tune, so we don’t know yet how far that alone moves the profile. Then again, if the profile moved, your old red-team result is probably stale either way.
Retrain a model on the exact same data, with the exact same safety training, and it falls for different jailbreaks.
We trained 150 models at 1B parameters (25 pretraining datasets × 3 pretraining seeds × 2 safety-training seeds) and tested each one on 3,212 jailbreak prompts. Changing only the pretraining seed matters almost as much as changing the data. Two safety-tuned copies of the same pretrained model agree at r = 0.98 on which prompts break them. That drops to 0.81 across pretraining seeds and 0.78 across datasets.
- The best attack on one checkpoint is still the best after a new pretraining seed only 27% of the time, vs. 67% after a new safety-training seed.
- 50 random prompts are enough to tell which of 75 pretraining runs a model came from, 98% of the time.
A jailbreak result describes one training run more than the data behind it. Code, weights, and per-prompt labels are released.
Models citing this paper 27
Browse 27 models citing this paperDatasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper