No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from AtomicChat/Ornith-1.5-35B-A3B-GGUF
All models by AtomicChat →Built from Ornith's original weights with our own importance matrix. The calibration corpora and raw eval logs behind this card are public.
GGUF builds of ornith-ai/Ornith-1.5-35B-A3B, converted and measured by AtomicChat.
A3B means 3B parameters are activated per token out of 35B total. Speed follows the 3B, memory follows the 35B: every expert has to be resident because the router picks a different eight of the 256 on every token.
[!TIP] With a smaller card, keep the experts in system RAM instead of dropping to a lower rung. Experts are 93% of the weights but only 3B of them run per token, so the arithmetic stays cheap while the bandwidth cost moves to RAM. Pass
--cpu-moeto put every expert on the CPU, or--n-cpu-moe Nto offload the experts of N layers and keep the rest on the GPU. That lets a 12 GB card serve a 22 GB build at usable speed.
Mean KL divergence against the BF16 reference, held-out eval_neutral corpus,
4096 context, llama.cpp b10505. Lower is better. Top-1 is the share of tokens
where the quant picks the same next token as the full precision model.
AD marks a per tensor bit layout tuned for this model. Files without the
prefix are stock llama.cpp presets built with the same importance matrix, so the
comparison isolates the layout and nothing else. Each AD label reads as
down_type-gate_up_type, which is what is actually inside the file.
Two rungs beat a stock preset on both axes at the same time.
AD-Q5_K-Q4_K is 2.6 GB smaller than stock Q5_K_M and more accurate.
AD-Q6_K-Q5_K is 2.3 GB smaller than stock Q6_K and more accurate. Neither
needs an interpolated curve to read off the chart.
At four bits the gap is largest. AD-Q4_K-IQ4_XS is a gigabyte smaller than
stock Q4_K_M and cuts divergence by 34%.
AD-Q6_K is within 12% of Q8_0 at almost eight gigabytes less. If you
were reaching for Q8_0 out of caution, this is the file to take instead.
The layout was not carried over from another model. We built five candidate layouts at the four bit size class and three more at six bits, measured all of them against the same reference, and kept the winners. Every log is in the metrics repo, including the ones that lost.
The winning layout is different at four bits and at six. At four bits,
lifting ffn_down_exps one step and paying for it out of ffn_gate_exps and
ffn_up_exps wins clearly. At six bits the same trade loses to a flat layout:
29.10 GB at 0.012961 for flat against 28.85 GB at 0.015291 for the asymmetric
one. The reason is that a coarse base type leaves a lot of headroom to
redistribute, while a fine one does not: lifting a small group buys nothing once
it is already near lossless, and the large group you took the bits from still
pays full price. The same inversion showed up independently on the dense
Ornith-1.5-9B, which makes it a property of quantization rather than of one
model.
ffn_down_exps is the sensitive one, and its shape says why. Its rows are
512 wide against 2048 for gate and up. A k-quant stores one scale per 256 wide
superblock, so a down projection row holds two superblocks while a gate row
holds eight. Less data per scale means a worse fit, so the extra bit pays off
there and nowhere else.
Cutting the non-expert weights does not pay. Experts are 93% of this model, so the attention, embedding and output matrices look like free real estate. We tested it: dropping all of them from q8_0 to q6_k saved 0.55 GB and cost 6.6% of the divergence, while spending the same 0.55 GB on the experts buys about 5.8%. The trade is a wash at best. Everything outside the experts stays at q8_0 in every file here, and the router stays in f32.
A sparse MoE costs more bits than a dense model of similar quality. Q8_0
here sits at 0.011620 while Q8_0 on the dense Ornith-1.5-9B reaches 0.002249,
five times lower, despite both having the same headroom. Short expert rows are
part of it. The router is the other part: it turns a small weight perturbation
into a hard switch to a different expert, so the error stops being smooth.
The measurements reproduce. The iq4_xs down with iq3_s gate and up
recipe was built and measured twice, on two different rented machines, and
returned mean KLD 0.057726 both times.
If the file does not fit your card, keep the experts in RAM rather than dropping a rung:
With vision, add the projector:
[!IMPORTANT] Pass
--image-min-tokens 1024. Without it, dense images such as charts and screenshots get too few visual tokens, and the model answers from what it already knows rather than from what it was shown. llama.cpp prints a warning about this at load time.
[!IMPORTANT] Upstream sampling defaults are not what llama.cpp falls back to. Set them yourself:
--temp 0.6 --top-p 0.95 --top-k 20. Use--temp 1.0to reproduce the benchmark setup from the upstream model card.
Lines reading find_slot: non-consecutive token position are normal. That is how
this family numbers image patches, and they appear in successful runs too.
Unlike the 9B, this checkpoint does ship a multi token prediction head, and we publish it as a separate draft file:
Note that -np > 1 and --mmproj do not currently work together with MTP in
llama.cpp, so pick either speculation or vision, not both.
Read from the converted file, not from the config.
Where the weights actually sit:
[!WARNING] If you convert this model yourself, pass
--no-nextntoconvert_hf_to_gguf.pyfor the main file and--mtpon a second run for the draft. Without the flag the MTP block lands inside the target file, where a plain forward pass never executes it, and low bit quantization aborts on it because the importance matrix has no statistics for a block that never runs.
The importance matrix was computed on the BF16 weights, not on a quantized stand-in, over 4,958,490 tokens in 9,686 chunks, fanned out across 15 GPUs and merged. The tokenizer is byte identical to Ornith-1.5-9B, so the same corpus build applies to both models.
Corpus composition: agentic tool traces 24.7%, code 17.8%, reasoning 14.8%, multilingual 13.8%, long context 11.9%, vocabulary sweep 9.9%, structured data 3.9%, graphics 3.0%. Vocabulary coverage is 99.5%.
For a sparse MoE the coverage question is sharper than for a dense model: a routed expert that never fires during calibration gets quantized blind, and at two bits that produces a file which loads and answers with nothing to warn you. At 8 of 256 experts per token, each expert sees roughly 155,000 tokens of this corpus, and the merged statistics show no expert tensor without data.
ornith-1.5-9b.The corpus and the context have to match ours or the numbers are not comparable.
Ornith-1.5 is trained with a loop that generates its own tasks, builds its own scaffolds, and rewards all three stages together rather than optimizing a policy against a fixed human written harness. For the method and the benchmark results, see the upstream model card and the Ornith blog. Those benchmark numbers are the authors' own and we have not independently reproduced them.
llama-server -m Ornith-1.5-35B-A3B-AD-Q4_K-IQ4_XS.gguf \ -ngl 99 -c 8192 -fa on --jinja \ --temp 0.6 --top-p 0.95 --top-k 20 \ --port 8080llama-server -m Ornith-1.5-35B-A3B-AD-Q5_K-Q4_K.gguf \ -ngl 99 --cpu-moe -c 8192 -fa on --jinja \ --temp 0.6 --top-p 0.95 --top-k 20 \ --port 8080llama-mtmd-cli -m Ornith-1.5-35B-A3B-AD-Q4_K-IQ4_XS.gguf \ --mmproj mmproj-Ornith-1.5-35B-A3B-F16.gguf \ -ngl 99 -c 8192 --temp 0.6 --image-min-tokens 1024 \ --image your-image.jpg \ -p "Describe this image."llama-server -m Ornith-1.5-35B-A3B-AD-Q4_K-IQ4_XS.gguf \ -ngl 99 -c 8192 -fa on --jinja -np 1 \ --spec-type draft-mtp --spec-draft-n-max 6llama-perplexity -m your-quant.gguf -f eval_neutral.txt \ --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99