We Fact-Checked Our Own Security Dataset. Here's What We Fixed.

*A July 2026 hardening pass on the SecureCode dataset family, its paper, and its models.

When we published SecureCode, the pitch was grounding. The paper said it plainly: "Every example ties to actual documented security incidents with CVE references." We believed it. Then we went back and checked every claim, one at a time, and found that a large fraction of them did not hold up.

This post is about what that audit found and what we did about it. The short version: the code lessons were sound, but the grounding claims were overstated, and a handful of "secure" examples did not actually fix the vulnerability they were teaching. We verified the real references, corrected or relabeled the rest, removed the examples that taught the wrong thing, and rebuilt the whole family so the viewer, the splits, and the metadata are trustworthy. We chose honesty over volume, and the dataset is stronger for it.

If you train on SecureCode, nothing about how you load it changes. The numbers moved a little and the grounding language is now accurate instead of aspirational.

What SecureCode is

SecureCode is an open, multi-turn dataset for teaching code models to write secure code. Every example pairs a realistic developer request with a vulnerable implementation, an explanation of why it is dangerous, a production-grade secure alternative with layered defenses, and a follow-up turn on testing and monitoring. It ships in three forms:

License is CC BY-NC-SA 4.0. It is defensive-security material: every vulnerable pattern is shown next to the fix.

The four things we checked

We ran the family through four independent audits. Each one was set up as an adversarial review: agents were told to try to refute a claim, and a claim only survived if it could not be refuted. Everything below is the result after that verification pass, not a first-pass guess.

1. Grounding integrity: are the CVEs and incidents real?

This was the big one, because "complete incident grounding" was the headline claim.

Web. We re-verified all 1,379 CVE and incident references against public sources. Most did not survive. We removed 802 mis-attributed CVEs (21 of them replaced with the correct CVE for the vulnerability shown), and we corrected 810 incident narratives: 378 were rewritten to match the real underlying event, and 432 were relabeled honestly as representative examples that are not tied to a specific public incident. When a CVE reference went away, any CVSS or EPSS score that had been harvested from that wrong CVE went with it, so the remaining scores actually describe the vulnerability on the page.

AI/ML. The pattern was the same, and worse. Of 747 examples, 613 carried "real world example" metadata that read like a breach report but was a representative scenario dressed up with invented precise statistics, exact dollar costs, exact record counts, unnamed victims, presented as if documented. We corrected all of them: 14 were tied back to the real incident they described (for example, Johann Rehberger's ZombAIs work and the Invariant Labs MCP tool-poisoning research), and the rest were relabeled as representative scenarios.

Here is the part that matters for anyone who trained on this: the fabrications were in the metadata field, not in the teaching. The CVEs cited inside the actual conversations were real. We verified them by hand, including EchoLeak (CVE-2025-32711), EmailGPT (CVE-2024-5184), the MCP inspector RCE (CVE-2025-52573), and the WebDAV zero-day (CVE-2025-33053). The code lessons never depended on the invented statistics.

The datasets now say what is true. Every dataset card carries a grounding note that states exactly what fraction of examples cite a documented incident versus a labeled representative scenario. No more "complete incident grounding."

2. Fix-correctness: does the "secure" code actually fix it?

A security dataset has one unforgivable failure mode: the example labeled "secure" is still exploitable. A model trained on that learns the wrong lesson with full confidence. So we reviewed every original example for whether the secure version actually eliminates the stated vulnerability, and adversarially verified every high-severity finding before acting on it.

Web. Reviewing all 1,412 pre-expansion examples surfaced 25 confirmed defects, zero disputed. Fourteen were mislabels, mostly a Spring Boot cluster where the developer asks for password hashing but the answer is about a different vulnerability entirely, incoherent across turns. Seven were genuinely dangerous: the "secure" version still shipped the bug, including Angular bypassSecurityTrustHtml, Jackson polymorphic typing left enabled, unfixed XXE and TOCTOU and IDOR, and an authorization check broken by operator precedence. Four fixed something other than the stated bug. All 25 were removed.

AI/ML. The same sweep across all 750 examples found three that were removed. The clearest: an example on redacting secrets from training data whose SecretScanner computed Shannon entropy but always returned 0.0, so the redaction never fired. It looked defensive and did nothing. The other two logged the very prompt they claimed to protect, and gated a knowledge-graph traversal behind access-control code that was dead on the live path.

We removed rather than patched because the point of this pass was to make every remaining example one you can trust without re-checking.

3. Leakage: is the test set clean?

Near-duplicate examples split across train and test inflate benchmark numbers. We built duplicate families using first-user-turn similarity (Jaccard over 0.75), kept each family intact within a single split, and stratified by category and language. Test-set contamination went from 11.6% to zero. Sizes and content were unchanged by the re-split; only the assignment of near-duplicates moved.

4. Viewer, schema, and metadata hygiene

The Hugging Face viewer was dead on two of the three datasets. Root cause: heterogeneous field types across examples broke Arrow's schema inference and segfaulted the parquet conversion. We now publish each config as parquet under one explicit shared schema, which fixed the viewer, search, filtering, and column statistics across all three. We also backfilled metadata.tags on every web example (language, category, OWASP codes, CWE, framework) so coverage is filterable from metadata, regenerated the taxonomy from the released data so it describes what is actually there, and corrected the schema documentation.

What also grew

Hardening was not only subtraction. While the corpus was open we added current, well-grounded coverage where it was thin: a server-rendered XSS pack, CSRF and open-redirect and clickjacking, file-upload and path-traversal, a Django buildout, and a currency pass on 2025-era issues including supply-chain and CI/CD security (the new OWASP 2021 to 2025 A03 category), OAuth and OIDC per RFC 9700, and recent framework CVEs such as the Next.js middleware bypass (CVE-2025-29927). Every CVE-citing example was checked against an allowlist of references we verified as real before ingest, and every new pack's complete code was syntax-checked and duplicate-checked at ingest.

Where the numbers landed

Web splits are 1,249 train / 186 validation / 190 test. The web corpus is now verified end to end: the entire original set was reviewed for fix-correctness, and every expansion pack was validated at ingest. AI/ML grounding and fix-correctness are both audited. All three viewers are live.

A note on the paper

The SecureCode paper (arXiv:2512.18542) has been revised to correct exactly the claims this audit uncovered. The revised version re-verifies every reference, reports grounding honestly instead of asserting it, documents the fix-correctness and leakage corrections, updates the counts to the current 2,372-example family, and adds a dedicated audit section walking through what changed and why. Read it for the full methodology and training setup; if you land on an older version, treat its grounding and leakage language as superseded by the revision, this post, and the dataset cards.

The models, retrained

Fixing the dataset is only half the point. Models trained on the old version had learned from the examples we removed, including "secure" code that did not actually eliminate the vulnerability, so the fix had to reach the weights too. We retrained the entire SecureCode model family on the audited 2,372-example set. The family now spans ten open models from 3B to 26B parameters: the refreshed Llama 3.2 3B, Qwen2.5-Coder 7B and 14B, DeepSeek-Coder 6.7B, CodeGemma 7B, CodeLlama 13B, StarCoder2 15B, and Granite 20B, plus two new Gemma 4 variants, the E4B edge model and the 26B mixture-of-experts. The refresh ran locally on a single NVIDIA DGX Spark GB10 (Grace Blackwell), using bf16 LoRA rather than the 4-bit QLoRA of the original A100 runs. Each model card records the change and links the corrected dataset, so anyone pulling a model can see exactly what it was trained on.

The principle

The temptation with a dataset is to defend the number. We had 2,185 examples and a clean-sounding grounding claim, and the easy path was to leave it alone. The audit made that impossible without lying, so we did the other thing: verified what was real, relabeled what was representative, removed what was wrong, and wrote down exactly where the line is. A slightly smaller dataset whose every claim you can check beats a larger one you have to take on faith. That is the version we want people training on.

Use it

from datasets import load_dataset

# Everything (2,372 examples)
ds = load_dataset("scthornton/securecode")

# Just web (1,625) or just AI/ML (747)
web  = load_dataset("scthornton/securecode", "web")
aiml = load_dataset("scthornton/securecode", "aiml")

• Unified: huggingface.co/datasets/scthornton/securecode
• Web: huggingface.co/datasets/scthornton/securecode-web
• AI/ML: huggingface.co/datasets/scthornton/securecode-aiml
• Models trained on SecureCode: huggingface.co/collections/scthornton/securecode

Questions, corrections, or a reference we still have wrong? Open a discussion on any of the dataset repos. Given what this audit was about, we mean that.