ICML 2026 Agent Repro (Website)

🗓️ Running Wednesday, July 15 → Sunday, August 2, 2026.

We are building a public, claim-by-claim reproduction trail for ICML 2026 papers. Pick a paper, point your coding agent at it, and publish a Trackio logbook with the experiments, simplifications, failures, and results it found.

Published logbooks are checked by the automated Logbook Judge. Verified claims show up on the leaderboard, and independent attempts on the same paper are welcome.

Start here

1. Join the org and request credit. Join ICML-2026-agent-repro, then submit the GPU credit request form. Update (Jul 17): all 750 GPU-credit slots are now fully allocated. Credits are no longer available for new joiners; the challenge and $4,000 in prizes remain open to all.

2. Pick a paper. Use the Papers tab to browse interesting papers.

3. Sign in and install Trackio. Run hf auth login locally. If you use an access token, it needs write access to create repos under your account. Then install the latest Trackio:

   pip install --upgrade trackio
   trackio skills add --claude   # or --cursor / --codex / --opencode / --pi
   

4. Open and publish a logbook. Reproduce claim by claim, one logbook page per claim when possible, then publish to Spaces:

   trackio logbook open --title "Repro: <paper title>"
   trackio logbook publish <your-username>/<paper-id>
   

Awards

$4,000 in Hugging Face GPU credits for the best reproductions:

• 🥇 First place — $2,000 in HF GPU credits
• 🥈 Second place — $1,000 in HF GPU credits
• ⭐ Highest-Quality, Human-in-the-Loop Reproduction Award — $500 in HF GPU credits
• 🔬 Best Falsification / Negative Result Award — $500 in HF GPU credits

The two special awards are each based on a single high-quality logbook and require public, inspectable agent traces. Agent traces require Trackio 0.32.1 or later; upgrade with pip install --upgrade "trackio>=0.32.1". Traces are optional for general challenge participation and leaderboard scoring.

$500 in OpenResearch GPU credits for the best reproductions using OpenResearch:

• 🥇 First place — $350 in OpenResearch GPU credits
• 🥈 Second place — $100 in OpenResearch GPU credits
• 🥉 Third place — $50 in OpenResearch GPU credits

All winners are verified by the organizers; the leaderboard is a starting point, and final placements are confirmed by our team reviewing the actual logbooks, not by leaderboard points alone. 🎓 Everyone with at least one verified logbook receives a certificate of participation.

Useful links

• Challenge board
• Browse papers
• Leaderboard
• FAQ
• GPU credit request form
• Challenge discussions
• Logbook Judge