Instructions to use RedHatAI/Qwen3.8-27B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/Qwen3.8-27B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RedHatAI/Qwen3.8-27B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("RedHatAI/Qwen3.8-27B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/Qwen3.8-27B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/Qwen3.8-27B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/Qwen3.8-27B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/Qwen3.8-27B-NVFP4

SGLang

How to use RedHatAI/Qwen3.8-27B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/Qwen3.8-27B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/Qwen3.8-27B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/Qwen3.8-27B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/Qwen3.8-27B-NVFP4
```

Qwen3.8-27B-NVFP4

Model Overview

Model Architecture: Qwen3_5ForConditionalGeneration
- Input: Text / Image
- Output: Text
Model Optimizations:
- Weight quantization: FP4 and FP8
- Activation quantization: FP4 and FP8
Release Date: 2026-09-21
Version: 2.0
Model Developers: RedHatAI

This model is an updated quantized version of Qwen/Qwen3.8-27B, using a mixed-precision FP4/FP8 scheme with an unquantized language-model head and updated quantization scales. See Evaluation for accuracy results.

Model Optimizations

This model was produced by applying mixed-precision quantization to Qwen/Qwen3.8-27B. MLP projections are quantized to FP4, attention projections and the final MLP layers are quantized to FP8, and the KV cache is quantized to FP8, while the language-model head is kept in full precision to preserve output quality. The quantization scales were updated by calibrating on a 512-sample subset of the perfectblend dataset with a recipe that combines AWQ and GPTQ.

Only the weights and activations of the linear operators within the transformer blocks are quantized using LLM Compressor. The checkpoint is ~24.7 GB on disk (versus ~54 GB in BF16), reducing disk size and GPU memory requirements by roughly 70%.

Deployment

vLLM Serving

vllm serve RedHatAI/Qwen3.8-27B-NVFP4 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"model":"RedHatAI/Qwen3.8-27B-speculator.dspark","num_speculative_tokens":8,"method":"dspark"}'

Creation

This model was created by applying LLM Compressor with calibration samples from perfectblend, as presented in the code snippet below.

from compressed_tensors.quantization.quant_scheme import (
    FP8_DYNAMIC,
    NVFP4,
    QuantizationScheme,
)
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier
from llmcompressor.utils import load_context

MODEL_ID = "Qwen/Qwen3.8-27B"

# Load model.
with load_context(Qwen3_5ForConditionalGeneration):
    model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)


recipe = [
    AWQModifier(duo_scaling="both"),
    GPTQModifier(
        config_groups={
            "attention": QuantizationScheme(
                targets=[
                    r"re:.*self_attn\.(q|k|v|o)_proj$",
                    r"re:.*linear_attn\.(in_proj_qkv|in_proj_z|out_proj)$",
                    r"re:.*layers\.(56|57|58|59|60|61|62|63)\.mlp\..*(gate|up|down)_proj$",
                ],
                **FP8_DYNAMIC,
            ),
            "mlp": QuantizationScheme(
                targets=[r"re:.*mlp\..*(gate|up|down)_proj$"],
                **NVFP4,
            ),
        },
        ignore=[
            "re:visual.*",
            "re:model.visual.*",
            "re:.*lm_head",
        ],
        kv_cache_scheme={
            "num_bits": 8,
            "type": "float",
            "symmetric": True,
            "strategy": "tensor",
            "dynamic": False,
            "observer": "static_minmax",
        },
    ),
]

# Apply quantization.
oneshot(
    model=model,
    processor=processor,
    recipe=recipe,
    dataset="perfectblend",
    splits="train[:512]",
    max_seq_length=4096,
    num_calibration_samples=512,
    moe_calibrate_all_experts=True,
)

# Save to disk in compressed-tensors format.
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-NVFP4-GPTQ-AWQ"
model.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)

Evaluation

This model was evaluated on GSM8K Platinum, MATH-500, AIME 2025, GPQA Diamond, and IFEval using lm-evaluation-harness (https://github.com/neuralmagic/lm-evaluation-harness) and lighteval (https://github.com/neuralmagic/lighteval), and on SWE Bench using Inspect AI (https://github.com/UKGovernmentBEIS/inspect_ai), served with vLLM (OpenAI-compatible API). Evaluations were run on 1x B200 GPU.

Accuracy

Recovery vs. BF16 baseline

Category	Benchmark	Qwen/Qwen3.8-27B	RedHatAI/Qwen3.8-27B-NVFP4	Recovery
Reasoning	GSM8K Platinum	96.25%	96.72%	100.49%
	MATH-500	83.67%	84.27%	100.72%
	AIME 2025	96.67%	95.00%	98.27%
	GPQA Diamond	89.56%	89.22%	99.62%
Instruction Following	IFEval	91.19%	91.99%	100.88%
Agentic - Coding	SWE Bench	78.8%	78.0%	98.98%

NVFP4 build comparison

Category	Benchmark	RedHatAI/Qwen3.8-27B-NVFP4	unsloth/Qwen3.8-27B-NVFP4	Inferact/Qwen3.8-27B-NVFP4
Reasoning	GSM8K Platinum	96.72%	95.42%	93.77%
	MATH-500	84.27%	85.67%	82.47%
	AIME 2025	95.00%	93.75%	91.66%
	GPQA Diamond	89.22%	89.39%	87.04%
Instruction Following	IFEval	91.99%	91.81%	91.50%

Reproduction

The results were obtained using the following commands. Each benchmark was run multiple times with different random seeds — 3 repetitions for GSM8K Platinum, MATH-500, GPQA Diamond, and IFEval, and 8 repetitions for AIME 2025 — and the reported score is the mean across seeds.

GSM8K Platinum & IFEval (lm-eval, 0-shot)

Run once per seed:

lm_eval --model local-chat-completions \
  --tasks gsm8k_platinum_cot_llama \
  --model_args "model=RedHatAI/Qwen3.8-27B-NVFP4,max_length=69632,base_url=http://127.0.0.1:3235/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_gsm8k_platinum.json \
  --seed 1234 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,max_gen_toks=32000,seed=1234"

lm_eval --model local-chat-completions \
  --tasks ifeval \
  --model_args "model=RedHatAI/Qwen3.8-27B-NVFP4,max_length=69632,base_url=http://127.0.0.1:3235/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_ifeval.json \
  --seed 1234 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,max_gen_toks=32000,seed=1234"

MATH-500, AIME 2025, GPQA Diamond (lighteval, 0-shot)

litellm_config.yaml:

model_parameters:
  provider: hosted_vllm
  model_name: hosted_vllm/RedHatAI/Qwen3.8-27B-NVFP4
  base_url: http://127.0.0.1:3235/v1
  api_key: ''
  timeout: 3600
  concurrent_requests: 32
  generation_parameters:
    temperature: 1.0
    max_new_tokens: 65536
    top_p: 0.95
    top_k: 20
    seed: 1234

Run once per seed (changing seed in the config each time):

lighteval endpoint litellm litellm_config.yaml 'math_500@1@3|0' --output-dir results/ --save-details
lighteval endpoint litellm litellm_config.yaml 'aime25@1@8|0' --output-dir results/ --save-details
lighteval endpoint litellm litellm_config.yaml 'gpqa:diamond@1@3|0' --output-dir results/ --save-details

Downloads last month: 45,718

Safetensors

Model size

20B params

Tensor type

BF16

F8_E4M3

Model tree for RedHatAI/Qwen3.8-27B-NVFP4

Base model

Qwen/Qwen3.8-27B

Quantized

(1214)

this model