No official torrent yet β requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from RedHatAI/Qwen3.6-35B-A3B-NVFP4
All models by RedHatAI β
This is a preliminary version (and subject to change) of NVFP4 quantized Qwen/Qwen3.6-35B-A3B model. The model has both weights and activations quantized to NVFP4 format with vllm-project/llm-compressor.
It is compatible and tested against vllm main. Deploy it with: vllm serve RedHatAI/Qwen3.6-35B-A3B-NVFP4 --reasoning-parser qwen3 --moe_backend flashinfer_cutlass.
If you have hardware with more compute than memory bandwidth, you may prefer this MoE variant for performance reasons.
Run this script with LLM Compressor main and latest transformers.
This model was evaluated on GSM8K-Platinum, MMLU-Pro, IFEval, Math 500, GPQA Diamond, AIME 25, and LiveCodeBench v6 using lm-evaluation-harness and lighteval, served with vLLM using --language-model-only.
The results were obtained using the following commands:
The model was served with vLLM using the following command:
Each benchmark was run 3 times with different seeds (42, 1234, 4158), except AIME 25 which used 8 seeds (42, 1234, 4158, 5322, 1356, 9843, 3344, 5678). Scores are averaged across all seeds.
litellm_config.yaml:
BFCL requires the model to be registered in the leaderboard codebase before running evaluation.
Step 1 β Register the model in bfcl_eval/constants/model_config.py
Add the following entry to api_inference_model_map:
Step 2 β Add the key to bfcl_eval/constants/supported_models.py
Add "Qwen3.6-35B-A3B-NVFP4" to the SUPPORTED_MODELS list.
Step 3 β Start the vLLM server (use the command at the top of this section; the --served-model-name flag ensures BFCL can find the model by its registered slug).
Step 4 β Generate responses and evaluate
import torchfrom compressed_tensors.utils import save_mtp_tensors_to_checkpointfrom datasets import load_datasetfrom transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifier
# NOTE: This example requires transformers >= v5
MODEL_ID = "Qwen/Qwen3.6-35B-A3B"
# Load model.model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")processor = AutoProcessor.from_pretrained(MODEL_ID)
# No need to include mtp layers as they are not loaded# through Qwen3_5MoeForConditionalGenerationrecipe = QuantizationModifier( targets="Linear", scheme="NVFP4", ignore=[ "re:.*lm_head", "re:visual.*", "re:model.visual.*", "re:.*mlp.gate$", "re:.*embed_tokens$", "re:.*shared_expert_gate$", "re:.*linear_attn.*", ],)
NUM_CALIBRATION_SAMPLES = 256MAX_SEQUENCE_LENGTH = 4096
ds = load_dataset( "HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]",)ds = ds.select_columns(["messages"])ds = ds.shuffle(seed=42)
def preprocess_function(example): messages = [ {"role": m["role"], "content": [{"type": "text", "text": m["content"]}]} for m in example["messages"] ] return processor.apply_chat_template( messages, tokenize=True, return_dict=True, add_generation_prompt=False, processor_kwargs={ "return_tensors": "pt", "padding": False, "truncation": True, "max_length": MAX_SEQUENCE_LENGTH, "add_special_tokens": False, }, )
ds = ds.map(preprocess_function, batched=False, remove_columns=ds.column_names)
def data_collator(batch): assert len(batch) == 1 return {key: torch.tensor(value) for key, value in batch[0].items()}
# Apply quantization.oneshot( model=model, recipe=recipe, dataset=ds, max_seq_length=MAX_SEQUENCE_LENGTH, num_calibration_samples=NUM_CALIBRATION_SAMPLES, moe_calibrate_all_experts=True, data_collator=data_collator,)
# Save to disk in compressed-tensors format.SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-NVFP4"model.save_pretrained(SAVE_DIR)processor.save_pretrained(SAVE_DIR)
# MTP layers are excluded from the model through Qwen3_5MoeForConditionalGeneration# Save them as-is from the original checkpoint into the quantized output.save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir=SAVE_DIR)vllm serve RedHatAI/Qwen3.6-35B-A3B-NVFP4 --reasoning-parser qwen3 --language-model-only --max-model-len 96000lm_eval --model local-chat-completions \ --tasks gsm8k_platinum_cot_llama \ --model_args "model=RedHatAI/Qwen3.6-35B-A3B-NVFP4,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=128,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results.json \ --seed 42 \ --gen_kwargs "do_sample=true,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=1.5,repetition_penalty=1.0,seed=42"lm_eval --model local-chat-completions \ --tasks ifeval \ --model_args "model=RedHatAI/Qwen3.6-35B-A3B-NVFP4,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=128,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \ --apply_chat_template \ --output_path results.json \ --seed 42 \ --gen_kwargs "do_sample=true,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=1.5,repetition_penalty=1.0,seed=42"lm_eval --model local-chat-completions \ --tasks mmlu_pro_chat \ --model_args "model=RedHatAI/Qwen3.6-35B-A3B-NVFP4,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=128,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \ --num_fewshot 0 \ --apply_chat_template \ --output_path results.json \ --seed 42 \ --gen_kwargs "do_sample=true,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=1.5,repetition_penalty=1.0,seed=42"model_parameters: provider: "hosted_vllm" model_name: "hosted_vllm/RedHatAI/Qwen3.6-35B-A3B-NVFP4" base_url: "http://0.0.0.0:8000/v1" api_key: "" timeout: 2400 concurrent_requests: 64 generation_parameters: temperature: 1.0 max_new_tokens: 64000 top_p: 0.95 top_k: 20 min_p: 0.0 presence_penalty: 1.5 repetition_penalty: 1.0 seed: 0lighteval endpoint litellm litellm_config.yaml \ "math_500|0,gpqa:diamond|0,lcb:codegeneration_v6|0" \ --output-dir results \ --save-detailslighteval endpoint litellm litellm_config.yaml \ "aime25|0" \ --output-dir results \ --save-details"Qwen3.6-35B-A3B-NVFP4": ModelConfig( model_name="Qwen3.6-35B-A3B-NVFP4", display_name="Qwen3.6-35B-A3B-NVFP4 (FC)", url="https://huggingface.co/RedHatAI/Qwen3.6-35B-A3B-NVFP4", org="Google", license="Apache 2.0", model_handler=OpenAICompletionsHandler, input_price=None, output_price=None, is_fc_model=True, underscore_to_dot=True,),bfcl generate --model Qwen3.6-35B-A3B-NVFP4 --test-category allbfcl evaluate --model Qwen3.6-35B-A3B-NVFP4 --test-category all