No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from RedHatAI/Qwen3.8-27B-MXFP4
All models by RedHatAI →This model is a quantized version of Qwen/Qwen3.8-27B. It was evaluated on several tasks to assess its quality in comparison to the unquantized model.
This model was obtained by applying GPTQ with AWQ activation smoothing via LLM Compressor, quantizing the weights and activations of the linear operators to the MXFP4 (FP4) data type and the KV cache to FP8, ready for inference with vLLM.
This optimization reduces the number of bits per parameter from 16 to 4 for the quantized weights and activations, reducing disk size and GPU memory requirements by approximately 75% (non-quantized components are kept in their original BF16 precision).
Additional model-specific options (long-context scaling, speculative decoding via the built-in MTP head, etc.) are documented in the vLLM recipe for Qwen3.8-27B.
This model was created by applying LLM Compressor with calibration samples from perfectblend, as presented in the code snippet below.
This model was evaluated on GSM8K Platinum, IFEval, MATH-500, AIME 2025, and GPQA Diamond using lm-evaluation-harness and lighteval, all served with vLLM (OpenAI-compatible API). Evaluations were run on 1× NVIDIA B200.
All tasks were evaluated 0-shot with seeds 1234/2345/3456 (3 repetitions); AIME 2025 used 8 seeds (1234/2345/3456/4567/5678/6789/7890/8901). The model was served with vLLM (the Deployment command) and the results were obtained using the following commands:
Run once per seed (changing --seed and the seed in gen_kwargs each time); use --tasks ifeval for IFEval.
litellm_config.yaml:
Run once per seed (changing seed in the config each time):
vllm serve RedHatAI/Qwen3.8-27B-MXFP4 \ --tensor-parallel-size 1 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xmlfrom transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.modifiers.transform.awq import AWQModifierfrom llmcompressor.utils import load_context
MODEL_ID = "Qwen/Qwen3.8-27B"
# Load model.with load_context(Qwen3_5ForConditionalGeneration): model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)processor = AutoProcessor.from_pretrained(MODEL_ID)
recipe = [ AWQModifier(duo_scaling="both"), GPTQModifier( targets="Linear", scheme="MXFP4", ignore=[ "re:visual.*", "re:model.visual.*", r"re:.*lm_head", "re:.*embed_tokens$", r"re:.*linear_attn\.in_proj_a$", r"re:.*linear_attn\.in_proj_b$", ], kv_cache_scheme={ "num_bits": 8, "type": "float", "symmetric": True, "strategy": "tensor", "dynamic": False, "observer": "static_minmax", }, ),]
# Apply quantization.oneshot( model=model, processor=processor, recipe=recipe, dataset="perfectblend", splits="train[:512]", max_seq_length=4096, num_calibration_samples=512, moe_calibrate_all_experts=True,)
# Save to disk in compressed-tensors format.SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-MXFP4"model.save_pretrained(SAVE_DIR)processor.save_pretrained(SAVE_DIR)lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=RedHatAI/Qwen3.8-27B-MXFP4,max_length=69632,base_url=http://127.0.0.1:3233/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --seed 1234 \ --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,seed=1234,max_gen_toks=32000" \ --apply_chat_template \ --output_path results_gsm8k_platinum.jsonmodel_parameters: provider: hosted_vllm model_name: hosted_vllm/RedHatAI/Qwen3.8-27B-MXFP4 base_url: http://127.0.0.1:3233/v1 concurrent_requests: 32 timeout: 3600 generation_parameters: temperature: 1.0 top_p: 0.95 top_k: 20 seed: 1234 max_new_tokens: 65536lighteval endpoint litellm litellm_config.yaml 'math_500|0' \ --output-dir results/ --save-details
lighteval endpoint litellm litellm_config.yaml 'aime25|0' \ --output-dir results/ --save-details
lighteval endpoint litellm litellm_config.yaml 'gpqa:diamond|0' \ --output-dir results/ --save-details