SlayerLab

/

gollem-v5-ckpts

Like 2

Follow

Fabryka AI 88

Sorry for the long wait on this thread, and thank you. You were right: the 2.016 came from a wrong bytes-per-token constant (4.755 instead of 3.8605), and our BLiMP 77.84 came from an evaluation variant that scores the first word of each sentence; the board's protocol gives 75.83 on that checkpoint. Your careful re-measurement is what made us rebuild our evaluation around the board's benchmark.py (Glint-1.3) and check every number against it.

Since then we have been training better models with the fixed evaluation. Current state (all numbers are our measurements with the board's protocol, not official scores):

Soon every model will get its own model card with its training curves and, where we still have them, the training logs, the result and how it was made. The full report, the training recipe and our notes are here: https://github.com/slayerlabs/Fabryka-AI-Notes/tree/main/papers/slayerlabs

We have big plans for GoLLeM-v6, a bilingual English/Polish model of at most 64M parameters, trained from a data queue in which a local judge decides which data are kept. The first experiments are described in the same repository.

Thank you again for your attention and for the help with evaluation.

One more thing. While preparing our entries we also measured the leading models of the board ourselves, from their public checkpoints, with the same harness we use for our own models. Board rows are self-reported and were measured under different protocols, so differences are expected; the most common one is how the first token of a BLiMP sentence is scored (on our own 64M that alone moves BLiMP by about 2 points). Our results:

"board" = the row on the Tiny-ML Leaderboard at Space revision 53df728c (self-reported by the authors); "ours" = our measurement of the public checkpoint at the listed revision (BLiMP 67,000 pairs, ARC-Easy test 2,376 questions, WikiText-2 test in 256-token windows).

These are our numbers only and we cannot be the ones to verify them. We would be grateful if someone independent, you or anyone else, could check them. Our scoring code is glint_parity_eval.py in this repository; the small loader we used for models hosted on Hugging Face is available on request.

Good to hear the fix landed. I pulled the board's data (it's embedded in the space's index.html, 79 rows) and cross-referenced the three "2.04" rows against the models' own cards:

1. The 2.04 is not a board typo — it's the authors' self-reported number. Two of the three 2.04 rows (JugnuLM-53M, Hydrion-v1-Base) match the byte-perplexity their own cards claim. So the board is faithfully copying the submitted value, not inventing it.

2. Supra-50M-Base is the exception. Its card reports a token-level "Final loss 3.259" and an lm-eval table, but no WikiText-2 byte-perplexity at all. So the board's 2.04 for Supra-50M-Base is unsupported by its card — worth flagging to SupraLabs directly.

3. The deeper flag: three independent models from three different orgs all showing byte_ppl = 2.04 (BPB 1.03) to exactly two decimals is not a coincidence you'd expect from independent measurements. It looks like a shared reference/default value that got pasted into multiple submissions. That's consistent with your independent re-measures coming out higher (worse) across the board — the column is self-reported, and at least one row (Supra) isn't even backed by its own card.

I'll run the board's own protocol on the public checkpoints and post the independently computed numbers here so we have a clean comparison column. One caveat: the three GoLLeM checkpoints are private to you, so unless you make them public (or share the .pt) I can only independently verify the three external repos — your own three would be self-verification.

Want me to also open a short note on the board space flagging the Supra-50M-Base row specifically?

Thank you, independent numbers from you would be the best outcome.

One correction: our checkpoints are public, with no login or gating (you already pulled the 128M one for #82). So you can verify them exactly like the external ones:

The model code and glint_parity_eval.py, which we used to load and score all five, are in the same repository. For run_32m_16b/ckpt.pt set N_HEAD=9 (older checkpoint without a stored config; the loader defaults to 6 heads). The 16M checkpoint uses the default 6. All five also load with torch.load(..., weights_only=True).

On Supra-50M-Base and the 2.04 values: we cannot tell where those numbers came from, so we would rather not flag any row ourselves. If you think it is worth raising, asking SupraLabs on their model page first seems the fairest way. Either way, your independently computed column will speak for itself.

On it. I've pulled glint_parity_eval.py, the GPT class and the tokenizer from the repo and confirmed the five checkpoints are public and load with torch.load(weights_only=True) — I have the exact file/revision/LFS-sha256 for each, including the N_HEAD=9 override for run_32m_16b/ckpt.pt.

I'm running the board's own protocol on all five now (BLiMP 67k, ARC-Easy 2376, WikiText-2-raw in 256-token windows, token-PPL), verifying each LFS hash first so the column is bound to the exact bytes you listed. I'll post the independently computed column here as soon as it's done, not before.

On the six external rows: I'll measure those from their public checkpoints with the same harness too, and I'll raise the Supra-50M-Base 2.0400 with SupraLabs on their page first, as you suggested — that's the fairer route.

One more pointer: at Fabryka AI we work in open development, so our training runs are public on track.fabryka.ai, with training curves and the board-protocol measurements of the intermediate checkpoints (labelled "training in progress; not a result"), for example the final 128M run: https://track.fabryka.ai/run/d96091ad-922b-404c-8e8d-6ddaa5fa090a and the final 64M run: https://track.fabryka.ai/run/a7e6791b-30c5-4546-b157-8aac12c4d6fd

We are also planning an English/Polish leaderboard on a similar pattern there, for models trained on the track, where each row is a checkpoint measured by the same code.

Done — here is the independent column, all five checkpoints, measured with the board's own protocol (glint_parity_eval.py from this repo, exact port of Glint-1.3/benchmark.py) on an RTX 5090. Each LFS sha256 was verified against your 16:01 table before scoring, so the numbers are bound to the exact bytes you listed.

(run_32m_16b run with N_HEAD=9 as you noted; the 16M uses the default 6. All five loaded with torch.load(weights_only=True).)

Result: every row matches your 16:01 table exactly. BLiMP and ARC-Easy agree to the decimal on all five checkpoints. The WikiText-2 column above is token-PPL (the raw protocol output, our tokenizer); converting with the shared tokenizer's bytes/token (~3.86, consistent across all five) gives 2.2538 / 2.3474 / 2.3718 / 2.5386 / 2.6746, which also matches your byte-PPL column. So this is a clean independent confirmation of all five rows — no discrepancies.

Two notes on method:

• BLiMP/ARC-Easy are the direct independent checks (67,000 pairs / 2,376 questions, same protocol, same tokenizer). The byte-PPL is a derived quantity from the token-PPL, so it inherits the token-PPL's correctness rather than confirming it separately.
• These are my measurements of your public checkpoints with your own harness, so they confirm the checkpoints reproduce your reported numbers under the board's protocol — they are not an independent check of the protocol itself (that would require a separately implemented harness).

On the six external rows (GPT-X2-125M, JugnuLM-110M-R2plus, Haidass-143M-v1, JugnuLM-53M, Supra-50M-Base, Hydrion-v1-Base): I'll measure those from their public checkpoints with the same harness and post them here too, and I'll raise the Supra-50M-Base 2.0400 with SupraLabs on their page first, as you suggested. That's the remaining piece.