No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from AtomicChat/Ling-3.1-flash-GGUF
All models by AtomicChat →GGUF builds of inclusionAI's Ling 3.1 flash with our own importance matrix. The calibration corpora behind our builds are public.
KL divergence is how far the quantized model's predictions drift from the
BF16 model's. Lower is better, and zero means identical. top-1 is how often it
picks the same next token the BF16 model would have picked. Both are measured
on held-out text; how is at the bottom.
The two 1-bit files are experimental: they matched the BF16 model's token choice 86 to 88% of the time here, so check them on your own tasks first.
AD- marks an Atomic Dynamic layout: the routed experts (97% of the weights)
take the low-bit type, while attention, the shared expert, the router and the
dense layers stay at q8_0 or higher. The file name is the type of the experts'
gate and up projections; their down projections, and all experts of the last
five blocks, get a step more.
The file has to fit, plus a little for the context. Context is cheap on this model: 63 of its 72 layers use linear attention with a fixed-size state, and only 9 keep a compressed attention cache. That is about 0.5 GB per conversation plus about 10 KB per token.
On a Mac, raise the GPU memory limit first, for example
sudo sysctl iogpu.wired_limit_mb=240000 on a 256 GB machine.
Each file sits in a folder of its name, split into shards of up to 45 GB; point llama.cpp at the first shard and it loads the rest:
The chat template is inside every file. Thinking is on by default; put
detailed thinking off in the system prompt to turn it off.
Reference: BF16 weights converted from inclusionAI's FP8 release and run unquantized (perplexity 2.3590 on the held-out set). Converting FP8 to BF16 rounds each weight once, far below what Q8_0 adds.
Held-out text: eval_neutral from our
calibration dataset,
never used for calibration, 144 chunks of 4096 tokens. Metric: per-token KL
divergence of each file's predictions against the reference, plus top-1
agreement. Hardware: 8x A100-SXM4-80GB, llama.cpp 097cf33d9 for every file.
The importance matrix is the one we computed on 5.5M tokens of the
ling3-moe corpus (thinking on and off, tool calls, 30 languages, code, long
documents up to 32k tokens), rendered through this model's own chat template.
Mixture of experts: 512 routed experts in each of 68 layers (the first 4 layers are dense), 8 routed plus 1 shared expert per token, about 24B parameters active of 564B. Hybrid attention: 63 Kimi Delta Attention (linear) layers and 9 multi-head latent attention layers. Hidden size 4096, vocabulary 157,184, context 262,144. The multi-token-prediction head of the release is not included.
Built from inclusionAI's FP8 release (revision b426279f), converted to BF16
and quantized with llama.cpp b11571 (097cf33d9); needs llama.cpp with
bailingmoe3 support.
MIT, as the original. Base model Ling 3.1 flash by inclusionAI. Quantization and evaluation by AtomicChat.
llama-server -hf AtomicChat/Ling-3.1-flash-GGUF:AD-Q4_K_M -ngl 99 -c 32768 --jinjahf download AtomicChat/Ling-3.1-flash-GGUF --include "AD-Q4_K_M/*" --local-dir .llama-server -m AD-Q4_K_M/Ling-3.1-flash-AD-Q4_K_M-00001-of-00009.gguf -ngl 99 -c 32768 --jinja