gemma-4-31B-it-FP8-dynamic returns only `<pad>` / empty output on vLLM 0.24.0

#7
by danbrima - opened

Environment

  • Model: RedHatAI/gemma-4-31B-it-FP8-dynamic
  • vLLM 0.24.0 on OpenShift, H100, TP4
  • Startup logs show CutlassFP8ScaledMMLinearKernel selected
  • No explicit --quantization flag
  • CPU offload disabled

Problem
Every request returns only <pad> tokens (token id 0), including a single plain-text request with no tools and no images. content and reasoning_content are both empty. The model generates the full 16,000 completion tokens and returns finish_reason: length.

Already ruled out

  • Removing --kv-cache-dtype fp8 did not help
  • Removing --reasoning-parser gemma4, --tool-call-parser gemma4 and --enable-auto-tool-choice did not help
  • vLLM #40290 (vision tower fp16 overflow) does not match, since this is text-only
  • vLLM #39392 (tool parser concurrency) does not match, since this reproduces on a single request with no tools

Expected behavior
Normal text completion, as documented on the model card.

Analysis
All-pad output on every request points to NaN or inf logits in the forward pass, with argmax falling to index 0. This does not match any existing upstream issue we found.

Sign up or log in to comment