No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from AtomicChat/Ling-3.0-flash-NVFP4
All models by AtomicChat →[!CAUTION]
THIS repo is broken! We're working on fix, sorry!
inclusionAI/Ling-3.0-flash quantized to NVFP4 in the compressed-tensors format, for vLLM and TensorRT-LLM.
76.9 GB across 16 shards. The GGUF builds for llama.cpp live in
AtomicChat/Ling-3.0-flash-NVFP4-GGUF.
Mean KL divergence against the bf16 baseline: 0.0409, top-1 agreement 95.9% - more accurate than our GGUF NVFP4 at a comparable size. Method and full numbers below.
Scheme NVFP4A16, as written into config.json:
4-bit E2M1 weights, one UE4M3 scale per 16 weights, plus a per-tensor global scale — tensor_group
is exactly the part the GGUF block layout has no room for, and it does put this build closer to the
original than our GGUF NVFP4 at the same bit width. Activations stay bf16, so no calibration data is
involved.
Same layout policy as the rest of our line: only the routed experts, 97.1% of the weights, are quantized. Everything cheap in parameters but expensive in error is left alone:
In the written config those patterns are expanded into an explicit list of more than 400 modules:
every attention.* projection (the KDA f_proj, b_proj, g_proj and the MLA kv_a_proj_with_mqa,
kv_b_proj, dense), every shared_experts.* and lm_head. Only the routed experts are in NVFP4.
The MTP block (layer 42) is kept here in bf16. The GGUF converter drops it, so if multi-token prediction lands in vLLM, this build has what it needs and the GGUF one does not.
Ling 3.0 is not in upstream vLLM yet, but the support is open as PR #51045 and can be installed straight from it, no need to wait for the merge:
The model authors also publish a fork, inclusionAI/vllm-ling-v3,
which works as well; the PR is closer to what will eventually ship upstream.
On arm64 machines such as DGX Spark, VLLM_USE_PRECOMPILED=1 will not help - those wheels are built
for x86_64 - so expect a full source build there.
NVFP4 has native tensor-core support on Blackwell (sm_100, sm_120, sm_121). Elsewhere it runs through dequantization, without the speedup.
NVFP4A16 is weight-only: activations stay bf16, so a model whose weights have been round-tripped
through the FP4 grid is mathematically the same model as this compressed checkpoint. That makes the
measurement easy to keep honest - the checkpoint is expanded back to dense bf16 and then goes through
the exact same harness as every other rung: llama-perplexity --kl-divergence against the same
bf16 baseline, the same 92 chunks of held-out text at 4096 tokens, the same 4x RTX PRO 6000 machine.
No second definition of the metric, no cross-runtime fudge factor.
The expansion was checked against compressed_tensors' own unpack_fp4_from_uint8 before use
(bit-identical) and against the original bf16 weights (relative L2 error 0.095 on the first tensor,
which is what FP4 costs).
Distribution of the divergence: median 0.0015, 99th percentile 0.448, maximum 21.9. As with every 4-bit build, the mean is carried by a small tail of tokens where the router flips to a different expert, not by a uniform blur.
Two things separate this build from the GGUF NVFP4 of the same width, and they pull in opposite
directions. In its favour, tensor_group carries a per-tensor global scale that the GGUF block has no
room for, and the ignore list keeps attention and the shared experts in bf16 rather than Q8_0. Against
it, the two dense FFN layers at the front of the model land in NVFP4 here, where the GGUF build keeps
them at Q8_0. On net it comes out 0.013 ahead.
Worth stating plainly: at 4-bit, NVFP4 is not the accuracy-per-byte winner in this line. AD-IQ4_XXS
is both smaller and closer to the original. NVFP4 is here for the Blackwell tensor cores, not for the
error curve.
Raw logs and json: AtomicChat/Ling-3.0-flash-GGUF-metrics.
format: nvfp4-pack-quantized | producer: compressed-tensorsgroup_0: bits=4 type=float strategy=tensor_group group=16 symmetric=Truetargets: ['Linear'] activations: NoneQuantizationModifier( targets=["Linear"], scheme="NVFP4A16", ignore=[ "lm_head", r"re:.*mlp\.gate$", # the router: an error changes which expert runs r"re:.*shared_experts\..*", # seen by every token r"re:.*attention\..*", # MLA, KDA projections and gates ],)git clone https://github.com/vllm-project/vllm && cd vllmgh pr checkout 51045 # or: git fetch origin pull/51045/head:ling3 && git checkout ling3VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
vllm serve AtomicChat/Ling-3.0-flash-NVFP4 --trust-remote-code --tensor-parallel-size 4