Instructions to use RedHatAI/gemma-4-31B-it-FP8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use RedHatAI/gemma-4-31B-it-FP8-dynamic with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="RedHatAI/gemma-4-31B-it-FP8-dynamic") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("RedHatAI/gemma-4-31B-it-FP8-dynamic") model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/gemma-4-31B-it-FP8-dynamic", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use RedHatAI/gemma-4-31B-it-FP8-dynamic with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "RedHatAI/gemma-4-31B-it-FP8-dynamic" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/gemma-4-31B-it-FP8-dynamic", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/RedHatAI/gemma-4-31B-it-FP8-dynamic
- SGLang
How to use RedHatAI/gemma-4-31B-it-FP8-dynamic with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "RedHatAI/gemma-4-31B-it-FP8-dynamic" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/gemma-4-31B-it-FP8-dynamic", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "RedHatAI/gemma-4-31B-it-FP8-dynamic" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/gemma-4-31B-it-FP8-dynamic", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use RedHatAI/gemma-4-31B-it-FP8-dynamic with Docker Model Runner:
docker model run hf.co/RedHatAI/gemma-4-31B-it-FP8-dynamic
gemma-4-31B-it-FP8-dynamic returns only `<pad>` / empty output on vLLM 0.24.0
Environment
- Model:
RedHatAI/gemma-4-31B-it-FP8-dynamic - vLLM 0.24.0 on OpenShift, H100, TP4
- Startup logs show
CutlassFP8ScaledMMLinearKernelselected - No explicit
--quantizationflag - CPU offload disabled
Problem
Every request returns only <pad> tokens (token id 0), including a single plain-text request with no tools and no images. content and reasoning_content are both empty. The model generates the full 16,000 completion tokens and returns finish_reason: length.
Already ruled out
- Removing
--kv-cache-dtype fp8did not help - Removing
--reasoning-parser gemma4,--tool-call-parser gemma4and--enable-auto-tool-choicedid not help - vLLM #40290 (vision tower fp16 overflow) does not match, since this is text-only
- vLLM #39392 (tool parser concurrency) does not match, since this reproduces on a single request with no tools
Expected behavior
Normal text completion, as documented on the model card.
Analysis
All-pad output on every request points to NaN or inf logits in the forward pass, with argmax falling to index 0. This does not match any existing upstream issue we found.