No official torrent yet β requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic
All models by RedHatAI β
This model is a quantized version of Qwen/Qwen3.5-397B-A17B. It was evaluated on several tasks to assess its quality in comparison to the unquantized model.
This model was obtained by quantizing the weights and activations of Qwen/Qwen3.5-397B-A17B to FP8 data type, ready for inference with vLLM.
This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor. Layers such as the visual encoder, linear attention (Gated DeltaNet), MoE router gates, shared experts, and token embeddings are kept in original precision.
This model can be deployed efficiently using vLLM.
Send requests to the server:
This model was created by applying LLM Compressor with FP8 dynamic quantization, as presented in the code snippet below.
This model was evaluated on GSM8K-Platinum, MMLU-Pro, IFEval, Math 500, GPQA Diamond, AIME 25, and LiveCodeBench v6 using lm-evaluation-harness and lighteval, served with vLLM using --language-model-only.
The results were obtained using the following commands:
The model was served with vLLM using the following command:
Each benchmark was run 3 times with different seeds (42, 1234, 4158), except AIME 25 which used 8 seeds (42, 1234, 4158, 5322, 1356, 9843, 3344, 5678). Scores are averaged across all seeds.
litellm_config.yaml:
vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --language-model-only --tensor-parallel-size 4vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --tensor-parallel-size 4vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --tensor-parallel-size 4vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' --tensor-parallel-size 4from openai import OpenAI
openai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8000/v1"
client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,)
model = "RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic"
messages = [ {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]
outputs = client.chat.completions.create( model=model, messages=messages,)
generated_text = outputs.choices[0].message.contentprint(generated_text)from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifier
MODEL_ID = "Qwen/Qwen3.5-397B-A17B"
# Load model.model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")processor = AutoProcessor.from_pretrained(MODEL_ID)
# Configure the quantization algorithm and scheme.# In this case, we:# * quantize the weights to fp8 with channel-wise quantization# * quantize the activations to fp8 with dynamic per-token quantizationrecipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=[ "re:.*lm_head", "re:visual.*", "re:model.visual.*", "re:.*mlp.gate$", "re:.*embed_tokens$", "re:.*shared_expert_gate$", "re:.*mlp\\.shared_expert$", "re:.*linear_attn.*", ],)
# Apply quantization.oneshot(model=model, recipe=recipe)
# Save to disk in compressed-tensors format.SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-FP8-dynamic"model.save_pretrained(SAVE_DIR)processor.save_pretrained(SAVE_DIR)vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --language-model-only --max-model-len 96000 --tensor-parallel-size 4lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=128,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results.json \ --seed 42 \ --gen_kwargs "do_sample=true,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=0.0,repetition_penalty=1.0,seed=42"lm_eval --model local-chat-completions \ --tasks ifeval \ --model_args "model=RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=128,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \ --apply_chat_template \ --output_path results.json \ --seed 42 \ --gen_kwargs "do_sample=true,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=0.0,repetition_penalty=1.0,seed=42"lm_eval --model local-chat-completions \ --tasks mmlu_pro_chat \ --model_args "model=RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=128,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results.json \ --seed 42 \ --gen_kwargs "do_sample=true,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=0.0,repetition_penalty=1.0,seed=42"model_parameters: provider: "hosted_vllm" model_name: "hosted_vllm/RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic" base_url: "http://0.0.0.0:8000/v1" api_key: "" timeout: 3600 concurrent_requests: 32 generation_parameters: temperature: 0.6 max_new_tokens: 64000 top_p: 0.95 top_k: 20 min_p: 0.0 presence_penalty: 0.0 repetition_penalty: 1.0 seed: 0lighteval endpoint litellm litellm_config.yaml \ "math_500|0,gpqa:diamond|0,lcb:codegeneration_v6|0" \ --output-dir results \ --save-detailslighteval endpoint litellm litellm_config.yaml \ "aime25|0" \ --output-dir results \ --save-details