Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
dejanseo 
posted an update 9 days ago
Post
100
10 Search Subqueries in 200 Microseconds: 1-Bit Consistency Fanout

We built Fanout Diffusion, an ultra-low-latency model that expands a search query into 10 diverse subquery vectors in a single forward pass.

Inspired by the continuous retrieval framework in R4T (arXiv:2603.06397), our goal was to make query expansion fast enough for production search without running language models.

How it works:
• Direct Embedding Space: Takes a query embedded via google/embeddinggemma-300m (768-dim) and predicts 10 distinct subquery vectors simultaneously.
• 1-Step Consistency Denoiser: Generates all 10 slots analytically in a single pass with zero ODE integration loops.
• 1-Bit Hardware Tensor Cores: Quantized ternary weights (-1, 0, +1) running on Ampere/Ada sub-byte PTX instructions with INT4 outer projections.

Key numbers on an RTX 4090:
• Latency: 0.200 ms (ONNX) / 0.329 ms (native C++ CUDA graph)
• Throughput: 417,035 queries/second in batch mode
• Model Size: 1.64 MB (PyTorch QAT) / 32.1 MB (ONNX graph)
• Quality: 0.683 prompt alignment across 540k search queries

Interactive Space:
dejanseo/fanout-diffusion

Model weights, ONNX graph, and C++ engine:
dejanseo/fanout-diffusion

Your own journal.db times the ONNX file at 2.43 ms, not 0.200.

I opened the tracker you shipped next to the weights. Three ONNX Runtime CUDA rows, one per export:

run                        onnx_latency_ms   max_diff_vs_pytorch
export-20260921-000538     2.339             0.000015
export-20260921-002927     2.600             0.000017
export-20260921-015428     2.430             0.011659

The last one is the file on the Hub. It logged 30.644230 MB, and the .onnx is 32,132,804 bytes, the same size to the byte.

The 0.200 looks like 1step_latency_ms: 0.2048 from Benchmark: Pure 1-Bit 4-Step & 1-Step CUDA Graph. That is a b1_tc run of the earlier pure_1bit_end_to_end model, not ONNX and not the QAT champion. The card's native 0.329 ms is latency_b1_ms of the Aligned 1-Step Consistency QAT training run. The two C++ rows say 0.169 ms and 0.541 ms at batch 1.

The engine can't time a query yet anyway. run_benchmark in fanout_engine.cu (the .exe prints the same strings) loops the 20 binary linears over a zero-filled buffer. LayerNorm and GELU are compiled in and never launched. The fp32 input, query and output projections are uploaded and never read. There is no attention kernel. So 417,035 QPS is 20 XOR-popcount GEMMs, not a fan-out.

The model itself checks out, and I think it is the better headline.

The ONNX is your 1.64 MB checkpoint exactly: all 20 binary matrices match the packed words bit for bit, and all 5 INT4 layers dequantize to identical floats. Every binary MatMul gets Where(x >= 0, 1, -1) on its input, so the activations are 1-bit too. A fully binarized denoiser holding your 97.5% alignment parity is a real result.

(One nit: 512 inputs pack into 16 uint32 words, so the weights are ±1. No zero state, so not ternary.)

Which number should the card carry for one real query: the 2.43 ms ONNX run, or a C++ run once the engine does attention?