No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from RedHatAI/Qwen3.5-4B-FP8-dynamic
All models by RedHatAI →This model is a quantized version of Qwen/Qwen3.5-4B. Evaluation results and reproduction steps are provided below.
This model was obtained by quantizing the weights and activations of Qwen/Qwen3.5-4B to FP8 data type, ready for inference with vLLM.
This optimization reduces the model weights from 9.3 GB to 8.0 GB on disk (~14% reduction). Activations are quantized dynamically at inference time using per-tensor scaling, requiring no calibration data.
Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor.
Multimodal (vision + text):
Text-only (lower memory):
This model was created by applying LLM Compressor using data-free FP8 dynamic quantization, as presented in the code snippet below.
llm-compressor==0.10.1.dev44+g437f8afecompressed-tensors==0.14.1a20260325transformers==5.3.0vllm==0.18.1lm-eval — neuralmagic/lm-evaluation-harness@741f1d8 (branch: mmlu-pro-chat-variant)lighteval — neuralmagic/lighteval@6f0f351 (branch: eldar-fix-litellm)This model was evaluated on GSM8k-Platinum, MMLU-Pro, IFEval, Math 500, AIME 2025, and GPQA Diamond using lm-evaluation-harness and lighteval, with inference served via vLLM.
The results were obtained using the following commands. GSM8k-Platinum, MMLU-Pro, IFEval, Math 500, and GPQA Diamond were each run 3 times with different seeds and results averaged. AIME 2025 was run 8 times. The vLLM server was started with --language-model-only for all evaluations.
Seeds used: 42, 1234, 4158
Seeds used: 42, 1234, 4158
Seeds used: 42, 1234, 4158
Seeds used: 42, 1234, 4158
Seeds used: 42, 1234, 1356, 3344, 4158, 5322, 5678, 9843
Seeds used: 42, 1234, 4158
vllm serve RedHatAI/Qwen3.5-4B-FP8-dynamic \ --reasoning-parser qwen3 \ --max-model-len 262144vllm serve RedHatAI/Qwen3.5-4B-FP8-dynamic \ --reasoning-parser qwen3 \ --max-model-len 262144 \ --language-model-onlyfrom openai import OpenAI
openai_api_key = "EMPTY"openai_api_base = "http://localhost:8000/v1"
client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,)
model = "RedHatAI/Qwen3.5-4B-FP8-dynamic"
messages = [ {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]
outputs = client.chat.completions.create( model=model, messages=messages,)
generated_text = outputs.choices[0].message.contentprint(generated_text)from compressed_tensors.utils import save_mtp_tensors_to_checkpointfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom transformers import AutoProcessor, AutoTokenizer, Qwen3_5ForConditionalGeneration
MODEL_ID = "Qwen/Qwen3.5-4B"
IGNORE_LAYERS = [ "re:.*lm_head", "re:.*embed_tokens$", "re:.*visual.*", "re:.*model.visual.*", "re:.*linear_attn.*",]
model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)processor = AutoProcessor.from_pretrained(MODEL_ID)
recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=IGNORE_LAYERS,)
oneshot(model=model, recipe=recipe)
model.save_pretrained("Qwen3.5-4B-FP8-dynamic", save_compressed=True)processor.save_pretrained("Qwen3.5-4B-FP8-dynamic")save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir="Qwen3.5-4B-FP8-dynamic")lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=RedHatAI/Qwen3.5-4B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results_gsm8k_platinum.json \ --seed <SEED> \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed=<SEED>"lm_eval --model local-chat-completions \ --tasks mmlu_pro_chat \ --model_args "model=RedHatAI/Qwen3.5-4B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results_mmlu_pro.json \ --seed <SEED> \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed=<SEED>"lm_eval --model local-chat-completions \ --tasks ifeval \ --model_args "model=RedHatAI/Qwen3.5-4B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=100,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results_ifeval.json \ --seed <SEED> \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0,max_gen_toks=65536,seed=<SEED>"lighteval endpoint litellm \ "model_name=hosted_vllm/RedHatAI/Qwen3.5-4B-FP8-dynamic,provider=hosted_vllm,base_url=http://0.0.0.0:8000/v1,timeout=3600,concurrent_requests=100,generation_parameters={temperature:1.0,max_new_tokens:65536,top_p:0.95,top_k:20,min_p:0.0,presence_penalty:1.5,repetition_penalty:1.0,seed:<SEED>}" \ "math_500@k=1@n=1|0" \ --output-dir results_math500 \ --save-detailslighteval endpoint litellm \ "model_name=hosted_vllm/RedHatAI/Qwen3.5-4B-FP8-dynamic,provider=hosted_vllm,base_url=http://0.0.0.0:8000/v1,timeout=3600,concurrent_requests=100,generation_parameters={temperature:1.0,max_new_tokens:65536,top_p:0.95,top_k:20,min_p:0.0,presence_penalty:1.5,repetition_penalty:1.0,seed:<SEED>}" \ "aime25@k=1@n=1|0" \ --output-dir results_aime25 \ --save-detailslighteval endpoint litellm \ "model_name=hosted_vllm/RedHatAI/Qwen3.5-4B-FP8-dynamic,provider=hosted_vllm,base_url=http://0.0.0.0:8000/v1,timeout=3600,concurrent_requests=100,generation_parameters={temperature:1.0,max_new_tokens:65536,top_p:0.95,top_k:20,min_p:0.0,presence_penalty:1.5,repetition_penalty:1.0,seed:<SEED>}" \ "gpqa:diamond@k=1@n=1|0" \ --output-dir results_gpqa_diamond \ --save-details