The expert did not stay the same size. It stayed the same count, and that took two moves that cancel.

Both configs are public, so this is checkable.

Alpamayo-1.5 expert: hidden 2048, FFN 8256, 16 heads. Alpamayo2-Super expert: hidden 1536, FFN 6144, 16 heads. The expert got 25 percent narrower.

Then the index files. model.safetensors.index.json carries expert.layers.0 through .35 on 1.5, and expert.expert.layers.0 through .63 on Super. Exactly the backbone depth in both cases, 36 and 64. Expert depth is not a design choice here. It is pinned to the backbone.

So the width cut is what pays for the depth increase. Per layer at those dims, 63.3M against 37.7M. Times 36 and 64: 2.28B to 2.42B. Both round to your 2.3B, but nothing was held fixed.

Which changes the distillation reading. The blocks are plain self-attention, no cross-attention tensors on either side, so the expert is not a head hanging off the end. It is a second tower interleaved one block per backbone block, and cache_layer_indices is null, so no subset. You cannot run the 2.3B expert without the 32B resident. Its cost is 64 extra blocks inside one forward pass, not a 2.3B model's cost.

And it means the expert does not survive a shortened backbone. Distill 64 layers to 32 and the expert's shape is invalid, so the compact part has to be retrained against the new teacher rather than carried across.

Which is the experiment I would want to see: has anyone kept a 64-layer expert against a shortened backbone, or does the pinning have to be relaxed first?

Checked the overhang and you are right, with one attribution I would move.

expert_update_cfg on Super is head_dim 128, hidden 1536, 16 heads. So attention projects 1536 up to 2048 and back, which is the dashed overhang. On 1.5 it is head_dim 128, hidden 2048, 16 heads, so 16 x 128 lands exactly on the hidden width and there is no overhang at all. It is new in this generation.

But the part that is pinned to the backbone is not the 2048.

Super's vlm_config.text_config is hidden 5120, FFN 25600, 64 query heads, 8 KV heads, head_dim 128, 64 layers. The expert's own llm_config is hidden 1536, FFN 6144, 16 query heads, 8 KV heads, head_dim 128, 64 layers.

What is shared is the KV side. 8 x 128 = 1024, identical on both towers, and that is the interface the expert has to match to read the cache. The 16 is the expert's own query count and it is not a backbone number, the backbone runs 64. Choose 12 heads instead and 12 x 128 = 1536 sits exactly on the hidden width, no overhang, and the KV interface is untouched. So the overhang buys 16 query heads, not backbone compatibility.

It is not free either. q_proj and o_proj each carry 512 of extra width, so 2 x 1536 x 512 per layer, 100.7M over 64 layers, 4.2% of the 2.42B.

The two towers also run different GQA ratios at the same head_dim, 2:1 on the expert against 8:1 on the backbone. Same KV width, four times the sharing on the big side.

Where this bites is the deployment line you drew, and it agrees with you.

Backbone per layer at those dims is 487.6M, so 64 layers is 31.2B, which is your 32B. Expert is 37.7M per layer, 2.42B, your 2.3B. The expert is 7.2% of the text stack today.

Now take the width prune you say is safe. Hidden 2560, FFN 12800, 32 query heads, KV geometry untouched. The expert carries over exactly as you describe, because nothing in it was sized to the backbone's width. The backbone drops to 7.97B.

The expert is now 23% of the model, not 7%.

So the prune that is safe for the expert is also the one that stops it being the compact part. "Remains at 2.3B" survives the prune and stops meaning what it meant before it.

Which is the number I would want in the deployment section: how far can the backbone be width-pruned before driving quality goes, given the expert reads that cache at every one of 64 layers and was trained against it at full width?