No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from AtomicChat/DeepSeek-V4-Flash-0731-GGUF
All models by AtomicChat →[!WARNING] The chat template inside these GGUF files is outdated and will silently degrade reasoning. It gates reasoning retention on
is_after_last_user, so in agentic loops every earlier turn's reasoning is dropped once tool results come back, andreasoning_effortnever reaches the model. Until the files are repacked, pass the corrected template explicitly:No re-download is needed — the template is a separate file at the repo root.
--jinjaalone is not enough: without--chat-template-filellama.cpp uses the stale template embedded in the GGUF. You do not needenable_thinkingor--reasoning on; thinking is on by default and the server injects it into the template context.
reasoning_effortacceptshighandmax. Clients that map every level tohighwill never reachmax— check your effort map.
DeepSeek-V4-Flash is a 284B parameter mixture of experts model (13B active per token): 43 layers, 256 routed experts with 6 active, one shared expert, 1M context, vocabulary 129,280. It is quantization-aware-trained — the official checkpoint already stores its routed experts in MXFP4 and everything else in FP8 or BF16.
The experts are 96% of the model and they are already 4.25 bits. There is nothing to gain above that: repacking MXFP4 into a wider format adds bytes and zero information. So AD-BF16 is not a "high precision" quant, it is the original weights in a GGUF container, verified byte for byte.
Everything below it trades expert bits for size, and because the model was trained at 4 bits, it has no spare precision to give. Errors below 3 bits grow faster than they would on a BF16-trained model. The table shows exactly how fast.
PPL — perplexity of the quant on wikitext-2. Mean KLD — how far its whole output distribution drifted from the lossless reference, in nats. Top-1 match — how often it picks the same next token as the reference; read this one if you read only one. Δp RMS — how much its confidence moved, in percentage points.
Names follow the real average bits per expert weight, not a size class. _XL means a rung sits above its tag. AD-IQ2_M at 2.79 bits carries the same expert budget that other publishers label IQ3_XXS.
Every number below was measured by us, on one harness, against one reference. Absolute values are not comparable to numbers published elsewhere — different corpora and context lengths shift them by tens of percent. Relative values within this table are.
Two pairs land on the same byte count — 96.8 GB and 104 GB — and we are ahead in both. The largest margin is at 118 GB, where AD-IQ3_XS is 19% closer to the reference than UD-IQ3_S for two extra gigabytes. Above 135 GB the difference is inside measurement error. Below 82 GB no other publisher ships a rung.
Pick the largest file that fits your memory with room for context.
DeepSeek V4 is supported by upstream llama.cpp — no PR checkout and no patching.
Build from master:
For AMD, swap -DGGML_CUDA=ON for -DGGML_HIP=ON -DGPU_TARGETS=<your gfx>.
[!IMPORTANT] Use
-fa off. With flash attention enabled, CUDA emits repeating<characters once the prompt exceeds a single pass (#26509). Quantized K-cache is also still broken (#25382) - leave the cache at its default type.
[!NOTE] This model declares a 1,048,576-token context.
llama-cliandllama-serverhonour that default and will try to allocate a KV cache for the full million tokens, which fails on any single GPU. Always pass an explicit-c.
[!NOTE] Earlier revisions of this card warned that the hyper-connection operations had no GPU kernels and that offload therefore gained little. That is fixed upstream: #25585 fused the hyper-connection ops and #25545 added a CUDA lightning indexer. On a build without them, sinkhorn normalisation and the hyper-connections decompose into thousands of elementwise kernels per token. Measured on one MI300X with
AD-IQ3_M, the same file went from 16.34 t/s to 37.27 t/s across that change — 2.28x, with no change to the model. Build from currentmaster.
convert_hf_to_gguf.py --no-lazy --outtype bf16. The --no-lazy flag is not optional — lazy evaluation corrupts token_embd.weight with NaN values on this architecture.conversion/deepseek.py so FP8-sourced tensors resolve to BF16 instead of the hardcoded Q8_0. Without this the base is not lossless: Q8_0 on FP8 tensors costs 0.219 KLD, because a linear int8 grid cannot represent a logarithmic float grid at the same bit count, and QAT trained the model to tolerate one and not the other.The base was verified against unsloth's independently produced lossless quant: top-1 agreement 100.000%, RMS Δp 0.000%, mean KLD within numerical noise.
Constant across every rung:
The experts vary. ffn_down_exps stays native MXFP4 on layers 41 and 42 in every rung; ffn_gate_exps and ffn_up_exps are split into the 18 highest-energy layers (20, 26–38, 40–42) and the remaining 25. The split comes from measured activation energy per layer via llama-imatrix --show-statistics, not from a hand-tuned map — energy rises monotonically with depth, from 45k at layer 0 to 661k at layer 42.
1,868,626 tokens, 229 chunks of 8,192. Vocabulary coverage 87.08% of 129,280 tokens seen at least once, p99 document length 12,251 tokens. Composition: graphics and 3D code 31.8%, long context 14.4%, agentic tool-call traces 13.8%, general code 12.8%, general text 9.0%, vocabulary sweep 8.3%, reasoning 5.8%, structured output 4.3%.
The vocabulary sweep exists because the first three layers route by a hash table keyed on token id rather than a learned gate — their expert coverage depends on vocabulary breadth, not corpus size. The 8,192 context exists because at short contexts the model's compressors and lightning indexer barely engage.
imatrix.dat is published at the repo root.
Reference is AD-BF16, corpus is the wikitext-2 test split from Salesforce/wikitext (parquet, concatenated), context 5,632, 51 chunks, llama.cpp build f180ae8b2.
Context 5,632 was chosen because perplexity of the lossless reference lands at 4.5289 there, matching the figure published for the official weights, and because at 512 tokens this architecture's indexer selects 512 of 512 and its compressors do nothing — the model is measured in a regime it was not built for.
llama.cpp has two code paths for MXFP4 weights. One unpacks them to BF16 and uses a
standard tensor-core matmul; the other feeds the packed 4-bit data straight into
block-scaled tensor-core instructions. The second is gated in ggml/src/ggml-cuda/common.cuh:
Compute capability is encoded as major*100 + minor*10, so the threshold of 1200 means
compute capability 12.0 — consumer Blackwell only. H100 and H200 are 9.0, B200 is 10.0,
B300 is 10.3. All of them fall below the threshold and take the dequantize path, despite
B200 and B300 having FP4 tensor cores in hardware.
Since the routed experts are 96% of this model and they are MXFP4, that branch changes the output distribution. Same file, same corpus, same context, same build commit:
That is a 4–5% gap from the GPU alone. Perplexity and KL-divergence numbers for this
model are not portable between GPU generations, which is why every number in this card
was measured by us on one machine — 8× RTX 5090, llama.cpp build f180ae8b2,
-DCMAKE_CUDA_ARCHITECTURES=120, -ngl 99 -c 5632 -b 5632.
Numbers published elsewhere were produced on different hardware; unsloth state 4× B200 for their table. Do not read across tables — compare within one.
All rungs above run on a single MI300X (192 GB HBM3, gfx942) with no tensor
parallelism and no CPU offload. Measured with llama-bench, ROCm 7.2.4,
llama.cpp with the fused DeepSeek V4 kernels, -ngl 99 -fa 0 -p 512 -n 128 -r 3.
Generation speed does not depend on quant size. Across a 1.4x spread in file size, token generation stays inside 36.92–37.75 t/s — a spread smaller than the run-to-run error. Prompt processing does vary, and there the larger IQ3 rungs are faster, because prefill runs real GEMMs and the dequantisation kernel differs per type.
The reason generation is flat is that this model is not memory-bound here. At 37 t/s with roughly 3.7 GB of active weights read per token, the effective bandwidth is about 137 GB/s against the 5.3 TB/s the card can deliver — 2.6% of it. What limits generation is kernel launch overhead across 43 layers, not memory traffic.
The practical consequence: pick the largest rung that fits your memory. On this hardware the bigger quant is free in generation speed and better in quality and prefill.
Peak VRAM was sampled during the run and includes weights plus working buffers
at the benchmark's context. Budget headroom on top for your own -c.
MIT, inherited from the original model. Copyright DeepSeek. Quantized and published by Atomic Chat.
curl -sLO https://huggingface.co/AtomicChat/DeepSeek-V4-Flash-0731-GGUF/resolve/main/chat_template.jinja
llama-server -m <your-quant>/...-00001-of-*.gguf \ --jinja \ --chat-template-file ./chat_template.jinja \ --chat-template-kwargs '{"reasoning_effort":"max"}' \ --reasoning-format deepseekgit clone https://github.com/ggml-org/llama.cpp && cd llama.cppcmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ONcmake --build build -j --target llama-cli llama-server./build/bin/llama-cli -hf AtomicChat/DeepSeek-V4-Flash-0731-GGUF:AD-IQ2_M \ --jinja --chat-template-file ./chat_template.jinja -ngl 99 -c 8192 -fa offpython3 -c "from huggingface_hub import hf_hub_downloadimport pandas as pdp = hf_hub_download('Salesforce/wikitext','wikitext-2-raw-v1/test-00000-of-00001.parquet',repo_type='dataset')open('wiki.txt','w').write(''.join(pd.read_parquet(p)['text']))"
./build/bin/llama-perplexity -m AD-BF16/DeepSeek-V4-Flash-0731-AD-BF16-00001-of-00004.gguf \ -f wiki.txt --kl-divergence-base ref.kld -ngl 99 -c 5632 -b 5632
./build/bin/llama-perplexity -m AD-IQ2_M/DeepSeek-V4-Flash-0731-AD-IQ2_M-00001-of-00004.gguf \ -f wiki.txt --kl-divergence-base ref.kld --kl-divergence -ngl 99 -c 5632 -b 5632#define GGML_CUDA_CC_BLACKWELL 1200#if __CUDA_ARCH__ >= GGML_CUDA_CC_BLACKWELL && __CUDA_ARCH__ < GGML_CUDA_CC_RUBIN# define BLACKWELL_MMA_AVAILABLE#endif