No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from pottokao/Qwen3.8-27B-NVFP4-MTP-2x16GB
All models by pottokao →A self-quantized NVFP4 build of Qwen3.8-27B with speculative decoding via the model's own MTP head, deployed on two consumer 16 GB cards (2× RTX 5070 Ti) with vLLM tensor parallelism — reaching a genuine 150,000-token context on current vLLM (160k on the older build this was developed against — see the version note). No A100/H100, no 48 GB card.
The point of this repo is the deployment recipe: what it takes to fit a 27B hybrid model + a 160k KV cache onto ~32 GB of split VRAM, on the cheapest hardware that can do it.
A 27B model normally wants a single 40–80 GB GPU. This build targets the opposite end: two ~16 GB desktop cards joined by tensor parallelism. Two things make that hard, and both are solved by the config below:
head_dim=256 on its 16 full-attention layers → 16,384 B/token/card, ~4.9× a typical model. A naive setup can't reach 128k, let alone 160k, on 16 GB cards.util × VRAM for weights + KV, but the attention workspace buffer is allocated outside that budget on the first request. At util=0.96 with the default 128 MiB workspace, the very first generation OOMs. Shrinking the workspace to 64 MiB is what lets utilization go high enough to reach 160k.Notes that will save you time:
--dataset cnn_dailymail is required. The 0.45 default calibration mix pulls a gated NVIDIA dataset and fails with DatasetNotFoundError.lm_head → NVFP4, KV → FP8.ignore: mtp*) — identical to nvidia/Qwen3.6-27B-NVFP4. A quantized MTP head does not load in vLLM.mtp, mtp_num_hidden_layers=1).See serve_tp2.sh. Core command:
The recipe above was developed against an earlier vLLM nightly. Re-running it
verbatim on v0.27.2rc1.dev110, on a second machine, surfaced two things:
-e NCCL_P2P_DISABLE=1 has been removed from the block above. It
belongs to a host whose GPU-to-GPU P2P is blacklisted by the driver, where vLLM
detects the lack of P2P and falls back on its own. On a host where P2P works,
vLLM still selects the CUSTOM all-reduce backend, then cannot obtain a peer
handle, and engine init dies with
Failed: Cuda error custom_all_reduce.cuh:164 'invalid argument'. Add it back
only if your host actually needs it.--gpu-memory-utilization, so 0.94 is
effectively 0.9164 (vLLM's own log suggests 0.9836 to restore the old
budget). At 0.94 the engine now refuses 160000 with "estimated maximum model
length is 150096". Either raise utilization or set --max-model-len 150000.Both were found by running this exact block, not by inspection. The numbers in
the next section come from the corrected 150000 variant.
Same benchmark suite, same machine and same day as the throughput section below,
so the two profiles are directly comparable. 8 categories × 16 prompts, 8 warmups,
--ignore-eos, thinking off, concurrency 1, --max-model-len 150000.
--ignore-eos forced-length generations on real
held-out prompts; clean short-form generation runs higher.Long context is nearly free here. The throughput profile below runs at 96k with an 8-slot scheduler and averages 114.8 tok/s on this identical suite — so trading 96k for 150k of context costs nothing measurable in single-stream speed (117.4 vs 114.8 is inside run-to-run noise on 7 of the 8 categories). What the long-context profile actually buys is 31% more KV pool, and what it gives up is batching headroom, not tokens per second.
Earlier revisions of this card quoted 123–128 tok/s and a 177,254-token pool. Those were measured on a different host and an older vLLM build and are not reproducible on current nightlies; the table above replaces them.
The section above is the latency profile: one stream, maximum context. This one is the opposite operating point — smaller context traded for concurrency, measured as a full sweep. Different flags, so the two are not interchangeable.
vLLM's own log states the constraint plainly: "Your GPU does not have native support for FP4 computation … Weight-only FP4 compression will be used leveraging the Marlin kernel." On sm_120 the NVFP4 win is footprint, not math throughput — weights land at 10.5 GiB/card, leaving room for the KV pool.
Concurrency 4 is the knee, and it is not close. Going 2→4 buys +120 tok/s. Going 4→6 buys +25 tok/s and triples time-to-first-token. Going 6→8 buys +12 tok/s for another 1.9× on TTFT. Past 4 you are paying latency for a rounding error.
Also worth knowing: --max-num-seqs 8 versus 4 is free and pointless —
measured at concurrency 4, seqs=4 scored 322.9 and seqs=8 scored 323.9 (0.3%,
noise), and the only cost was 4.2% of the KV pool. The ceiling is not the
scheduler slot count.
Three things fall out of this table:
coding has the worst
accept_len in the table (2.53) and the fourth-best throughput; agent has a
good one (3.06) and the worst throughput. Prompt shape dominates.The KV pool is 114,480 tokens total, shared. Eight concurrent streams means roughly 14k tokens of context each, not 8 × 98k. This profile serves eight short-context sessions or one long one — never both. If you need long context per session, use the long-context profile above and accept ~117 tok/s.
(One outlier for honesty: agent scored 188.9 at concurrency 6, below its own
227.7 at concurrency 4 — the only non-monotonic cell in the sweep. n=16 per
category is not enough to smooth that out; don't build a story on it.)
Throughput is worthless if the quantization made the model dumber, so this checkpoint was run against the full AIME 2026 — both papers, all 30 problems.
Hardware for this run: 2× RTX 5060 Ti, vLLM TP2 — cheaper cards than the 5070 Ti pair in the deployment section above, same 16 GB-per-GPU class.
pass@1 = 29/30 — first sample of each problem, single pass, no retries. any-correct = 30/30 — Q10 was resampled and answered correctly on retry. This is the weaker of the two metrics; quote pass@1.
That works out to 65.4 tok/s sustained across 2.5 hours of real reasoning, against 67.7 tok/s measured on the same cards with the short-prompt benchmark suite. The MTP speedup holds on long chain-of-thought — it is not an artifact of short benchmark prompts.
The single miss is Q10: 61,244 tokens, finish_reason: stop, answered 165
instead of 156. It burned more tokens than most of the problems it got right
and still went astray — a hard problem it over-thought, not a compute shortfall.
No generation in the run hit the token ceiling; all 30 ended with stop.
This is corroboration that the quantization did not damage the model — not a lossless claim.
What it rules out: the loud failure modes. Garbled output, broken logic, divergence on long generations, losing the thread mid-proof. A 4-bit checkpoint that reasons correctly for 2.5 hours across 580k tokens is not a lobotomized one.
What it does not rule out: a quiet 1–2 point cost at the top of the range. There is no BF16 control run on these same 30 problems, and Qwen's own model card publishes no AIME figure for this model (it reports GPQA Diamond, HLE, LiveCodeBench v6 and MathVision) — so there is no external anchor to subtract from either.
temperature 0.6,
29/30 means "one sample scored 29/30", not "solves 29 of 30 reliably".finish_reason: length and score as wrong — an artifact of the harness, not
of the model.Qwen/Qwen3.8-27B and retains its safety alignment.--language-model-only); this is a text deployment.vllm/vllm-openai:nightly (needs a build new enough to resolve Qwen3_5MTP and the mtp speculative method under TP2).nvidia/Qwen3.6-27B-NVFP4.Qwen/Qwen3.8-27B).cd modelopt/examples/llm_ptqpython hf_ptq.py \ --pyt_ckpt_path <Qwen/Qwen3.8-27B snapshot> \ --export_path ./qwen38-nvfp4 \ --recipe huggingface/qwen3_5/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast \ --dataset cnn_dailymail \ --trust_remote_codedocker run -d --gpus all \ -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ -e VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=67108864 \ -v /path/to/qwen38-nvfp4-mtp:/model -p 8000:8000 \ vllm/vllm-openai:nightly \ /model --tensor-parallel-size 2 --trust-remote-code --served-model-name qwen38 \ --language-model-only \ --kv-cache-dtype fp8 --attention-backend TRITON_ATTN \ --mamba-cache-dtype bfloat16 --mamba-ssm-cache-dtype bfloat16 \ --gpu-memory-utilization 0.94 \ --max-model-len 150000 \ --max-num-seqs 3 --max-num-batched-tokens 1024 --enable-chunked-prefill \ --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \ --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice