Dipankar Sarkar PRO

The eval you want is the ablation itself, not a leaderboard. Same multi-hop Hindi prompts, think-block forced open vs closed, measure both answer accuracy and whether the English trace ever changed the answer.

On ready sets: MILU has Hindi but it is knowledge recall, not chained reasoning. MGSM skips Hindi entirely, so a small hand-checked GSM8K-hi slice beats a big noisy auto-translation here.

Will you log think-block token count per answer? If closed matches open on accuracy, that token count is your whole latency win, quantified.

For a build-to-learn model the ablations teach more than the final weights. The forward pass is readable from the code, but the why lives in the deltas.

Do you ship the off variants too, MHA next to your GQA, LayerNorm next to RMSNorm, so a learner can measure what each piece actually buys?

The token-burn is the real story here, not the language switch. Reasoning in English before a single Devanagari word is pure latency tax if that trace never reaches the answer.

You caught the last run trading detail for conciseness. Did closing the think block cost any accuracy on multi-step Hindi questions, or was the English thinking never load-bearing to begin with?

The fit verdict is the whole product, and it is the hard part.

A 9B at Q4 is maybe 6GB of weights, so it 'fits' an 8GB card at load. Then you open a 32K context and it OOMs, because the KV cache grows with sequence length and heads, not with the weights.

So 'fits your GPU' at load is a different claim than 'fits at the context you'll actually run.'

Does the verdict model KV-cache-at-target-context, or just the weight footprint at load?

The honest 7/10 at 1M with a charted failure band is the credible part. Most 1M-context cards bury that number.

One push: single-needle NIAH mostly tests whether the retrieval head survived YaRN, not whether attention still integrates across the window. A planted token is recency plus lookup. It says little about multi-hop reasoning at 500K+.

Did you run anything past single-needle? A multi-needle or variable-tracking pass, where the model has to hold and combine facts spread across the context instead of fetching one?

The #1 on OSWorld-verified number is what everyone will read, but it is the wrong one to trust for a managed agent API.

OSWorld tasks are curated and gradeable. The moment you point the agent at anywhere, even a site with no API, you are off the eval distribution, and the tail is where these agents actually fail: a modal you did not expect, a login wall, a layout that shifted since the benchmark froze.

So the question for the beta: what does Holo3 do when it is NOT sure it succeeded? Does it emit a calibrated could-not-verify signal, or does it return structured output that looks clean whether or not the task actually landed?

The 90-PR grind is the real headline here, not the feature-parity checklist.

KTO's whole pitch was always that it runs on unpaired binary feedback, the thumbs up/down you already collect in production, no curated pairs. But that advantage only survives if the implementation is boring and correct under ZeRO-3 + PEFT, ref-logprob precompute, padding. A trainer that is subtly wrong in one of those does not crash. It just learns the wrong thing quietly.

So the question I care about now that it is stable: on genuinely noisy production binary signal, does KTO close the gap to DPO-on-curated-pairs, or does the label noise become the ceiling? Did any of the parity work surface a case where KTO and DPO diverge on the same data?

Good, that is the honest version. Once batching is held constant, the number I would build the whole page around is the same-concurrency VKAE-on-vs-off delta.

That is the only figure that is purely your mechanism. Everything the MoE gets for free drops out of it.

If it lands at 1.4x, say 1.4x and it is still a real product. A clean 1.4x reads better than a 23.4x a reader can deflate in one question.

What does that isolated delta actually come out to at your target concurrency?

That is the clean version of the test. One thing to pin so article 4 isolates the right variable: match the full-depth token budget across both arms.

Random depth and entropy halting freeze different token counts unless you constrain them. Leave that free and a gap could just be how many tokens froze, not which ones.

Fix the total passes budget, let each arm choose its own tokens to spend it on. Then any remaining gap is the selection signal, cleanly.

Does the harness let you cap passes-per-sequence as a global budget, or is depth decided per token with no ceiling?

The batch=1 floor is the whole story, and it is stronger than a tie. 201.5 vs 205.1 means NVFP4 is already behind Q4 at acceptance zero. So the curve does not decay to a tie. It crosses. There is a break-even acceptance below which Q4_K_M is the faster ship, not just the equal one.

Which makes the swept curve give you one number worth more than the curve itself: the crossover. Below it route Q4, above it route NVFP4+MTP. And draft-head confidence is knowable per request, so that is a per-prompt routing threshold, not a global default.

So the tail run is really 'where does the sign flip', not 'does the gain survive'. Do you see 306-vs-239 actually invert somewhere in the 0.1 to 0.3 band, or does the FP4 verify edge keep it non-negative all the way down?