Fix KV cache reuse in streaming moderation
The streaming helper currently recomputes the full growing prefix with KV caching disabled. This PR enables per-stream caching, processes only new tokens after prefill, and fixes single-token result extraction.
The current modeling_qwen3_guard.py is identical across the 0.6B, 4B, and 8B repositories, so the same source fix applies.
Validation: 12 tiny-model CPU regression tests passed. Real-weight correctness and performance tests were run on 0.6B only; this model size has not been benchmarked.
0.6B reference results: Apple M4 / 32 GiB, MPS float32 SDPA, PyTorch 2.8.0 / Transformers 4.55.4. Cached logits matched full-prefix and original-stream latest-token logits with max absolute error 3.94e-6 (32-token prompt + 4 new tokens; atol=rtol=1e-4).
| 0.6B metric (median) | Before | After |
|---|---|---|
| Incremental latency | 116.26 ms/token | 28.66 ms/token |
| Total, including prefill | 1958.92 ms | 516.25 ms |
Timing used a 64-token prompt + 16 new tokens, 3 trials after warmup, alternating order and MPS synchronization; loading and tokenization excluded. These end-to-end results include KV reuse and reduced historical-output processing.
API note: streaming outputs now contain only the latest token, matching the docstring, rather than the entire prefix. Moderation result fields remain one-element lists.
We've confirmed that this change brings a significant speedup. Thanks!