Cataloged from RedHatAI/Qwen3.5-9B-FP8-dynamic
This model is a quantized version of Qwen/Qwen3.5-9B. Evaluation results and reproduction steps are provided below.
This model was obtained by quantizing the weights and activations of Qwen/Qwen3.5-9B to FP8 data type, ready for inference with vLLM.
This optimization reduces the model weights from 19.3 GB to 14.0 GB on disk (~27% reduction). Activations are quantized dynamically at inference time using per-tensor scaling, requiring no calibration data.
Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor.
Multimodal (vision + text):
vllm serve RedHatAI/Qwen3.5-9B-FP8-dynamic \
--reasoning-parser qwen3 \
--max-model-len 262144
Text-only (lower memory):
vllm serve RedHatAI/Qwen3.5-9B-FP8-dynamic \
--reasoning-parser qwen3 \
--max-model-len 262144 \
--language-model-only
from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
model = "RedHatAI/Qwen3.5-9B-FP8-dynamic"
messages = [
{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]
outputs = client.chat.completions.create(
model=model,
messages=messages,
)
generated_text = outputs.choices[0].message.content
print(generated_text)
This model was created by applying LLM Compressor using data-free FP8 dynamic quantization, as presented in the code snippet below.
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from transformers import AutoProcessor, AutoTokenizer, Qwen3_5ForConditionalGeneration
MODEL_ID = "Qwen/Qwen3.5-9B"
IGNORE_LAYERS = [
"re:.*lm_head",
"re:.*embed_tokens$",
"re:.*visual.*",
"re:.*model.visual.*",
"re:.*linear_attn.*",
]
model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=IGNORE_LAYERS,
)
oneshot(model=model, recipe=recipe)
model.save_pretrained("Qwen3.5-9B-FP8-dynamic", save_compressed=True)
processor.save_pretrained("Qwen3.5-9B-FP8-dynamic")
save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir="Qwen3.5-9B-FP8-dynamic")
llm-compressor==0.10.1.dev44+g437f8afecompressed-tensors==0.14.1a20260325transformers==5.3.0vllm==0.18.1lm-eval — neuralmagic/lm-evaluation-harness@741f1d8 (branch: mmlu-pro-chat-variant)lighteval — neuralmagic/lighteval@6f0f351 (branch: eldar-fix-litellm)This model was evaluated on GSM8k-Platinum, MMLU-Pro, IFEval, Math 500, AIME 2025, and GPQA Diamond using lm-evaluation-harness and lighteval, with inference served via vLLM.
The results were obtained using the following commands. GSM8k-Platinum, MMLU-Pro, IFEval, Math 500, and GPQA Diamond were each run 3 times with different seeds and results averaged. AIME 2025 was run 8 times. The vLLM server was started with --language-model-only for all evaluations.
lm_eval --model local-chat-completions \
--tasks gsm8k_platinum_cot_llama \
--model_args "model=RedHatAI/Qwen3.5-9B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
--num_fewshot 0 \
--apply_chat_template \
--output_path results_gsm8k_platinum.json \
--seed <SEED> \
--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed=<SEED>"
Seeds used: 42, 1234, 4158
lm_eval --model local-chat-completions \
--tasks mmlu_pro_chat \
--model_args "model=RedHatAI/Qwen3.5-9B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
--num_fewshot 0 \
--apply_chat_template \
--output_path results_mmlu_pro.json \
--seed <SEED> \
--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed=<SEED>"
Seeds used: 42, 1234, 4158
lm_eval --model local-chat-completions \
--tasks ifeval \
--model_args "model=RedHatAI/Qwen3.5-9B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
--num_fewshot 0 \
--apply_chat_template \
--output_path results_ifeval.json \
--seed <SEED> \
--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed=<SEED>"
Seeds used: 42, 1234, 4158
lighteval endpoint litellm \
"model_name=hosted_vllm/RedHatAI/Qwen3.5-9B-FP8-dynamic,provider=hosted_vllm,base_url=http://0.0.0.0:8000/v1,timeout=3600,concurrent_requests=100,generation_parameters={temperature:1.0,max_new_tokens:65536,top_p:0.95,top_k:20,min_p:0.0,presence_penalty:1.5,repetition_penalty:1.0,seed:<SEED>}" \
"math_500@k=1@n=1|0" \
--output-dir results_math500 \
--save-details
Seeds used: 42, 1234, 4158
lighteval endpoint litellm \
"model_name=hosted_vllm/RedHatAI/Qwen3.5-9B-FP8-dynamic,provider=hosted_vllm,base_url=http://0.0.0.0:8000/v1,timeout=3600,concurrent_requests=100,generation_parameters={temperature:1.0,max_new_tokens:65536,top_p:0.95,top_k:20,min_p:0.0,presence_penalty:1.5,repetition_penalty:1.0,seed:<SEED>}" \
"aime25@k=1@n=1|0" \
--output-dir results_aime25 \
--save-details
Seeds used: 42, 1234, 1356, 3344, 4158, 5322, 5678, 9843
lighteval endpoint litellm \
"model_name=hosted_vllm/RedHatAI/Qwen3.5-9B-FP8-dynamic,provider=hosted_vllm,base_url=http://0.0.0.0:8000/v1,timeout=3600,concurrent_requests=100,generation_parameters={temperature:1.0,max_new_tokens:65536,top_p:0.95,top_k:20,min_p:0.0,presence_penalty:1.5,repetition_penalty:1.0,seed:<SEED>}" \
"gpqa:diamond@k=1@n=1|0" \
--output-dir results_gpqa_diamond \
--save-details
Seeds used: 42, 1234, 4158