No official torrent yet β requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from empero-ai/Qwythos-9B-v2-GGUF
All models by empero-ai β
Empero AI
GGUF quantizations of empero-ai/Qwythos-9B-v2 for llama.cpp, Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes.
Qwythos-9B-v2 is the new and improved Qwythos β all the deep chain-of-thought reasoning of the base Qwythos, with the looping behavior fixed. The looping/degeneration that showed up under greedy or low-temperature decoding is trained out (6.7% β 0%), the native MTP head is restored, and the identity prompt is cleaned up β while knowledge and reasoning are held at (or above) the base Qwythos level.
The fix uses FTPO (Final-Token Preference Optimization): the exact token that starts a repetition loop is identified and the model is gently trained to prefer coherent alternatives at that one position, leaving the rest of the distribution β and therefore its knowledge and reasoning β untouched.
For full training details, evaluation numbers, and sample generations, see the base model card.
repeat-penalty as a band-aid.-MTP- files, so speculative-decoding (--spec-type draft-mtp) works.If you don't know which to pick, Q4_K_M is the right starting point.
nextn_predict_layers = 1)These embed the restored Qwen3.5-compatible MTP head. Use them with llama.cpp builds that support MTP draft speculation (--spec-type draft-mtp). The MTP matrices are retained at Q8_0 in every quantized variant.
The vision tower is inherited unchanged from Qwen3.5-9B β it was frozen through both the base Qwythos SFT and the v2 FTPO fine-tune, so image behavior matches base Qwen3.5-9B. This mmproj is interchangeable with any Qwen3.5-9B mmproj-*.gguf.
Qwythos is a hybrid model β a 3:1 mix of Gated-DeltaNet linear-attention (SSM) blocks and full-attention blocks. The linear-attention state tensors are disproportionately sensitive to low-bit quantization, so the K-quants here keep them at higher precision than the surrounding weights:
The remaining SSM state tensors (ssm_a, ssm_conv1d, ssm_dt, ssm_norm) are kept at F32 by the converter. This preserves the hybrid/SSM blocks for a small (~2β4%) increase in file size over a flat K-quant. Q8_0 and BF16 are uniform and need no overrides.
Because v2's looping is trained out, --repeat-penalty is now optional and greedy decoding (--temp 0) stays coherent.
Drop any .gguf into your runtime's model directory. Qwythos uses the standard Qwen3.5 chat template; modern GGUF runtimes load it automatically from the file.
MTP support requires a recent llama.cpp build. If your runtime doesn't support MTP yet, use the normal files.
Download a text quant plus the mmproj-*.gguf, then run llama.cpp's multimodal CLI/server:
Honest note: all Qwythos training (base SFT and v2 FTPO) was text-only β the vision tower was never fine-tuned, so image-grounded reasoning inherits base Qwen3.5-9B behavior and has not been independently evaluated for this release.
Qwythos is a reasoning model β every response opens with a <think>...</think> block before the answer.
Unlike the base Qwythos, v2 does not loop under greedy / low-temperature decoding β you can use --temp 0 for deterministic runs without repetition. The 0.6-temperature settings above still match Qwen3.5's official thinking-mode recommendations for best quality.
The GGUFs ship with YaRN rope-scaling baked in for a 1,048,576-token context window (4Γ the 262,144 native). Set -c up to 1048576; lower it to reduce KV-cache memory for shorter prompts. A single H100/H200-class GPU comfortably handles 256kβ512k; the full 1M typically needs multi-GPU or aggressive KV-cache offload.
convert_hf_to_gguf.py, llama-quantize), architecture qwen35, GGUF v3.nextn_predict_layers = 1, 15 MTP tensors, MTP matrices pinned Q8_0). Normal variants: --no-mtp (32-block trunk-only). mmproj: --mmproj --outtype bf16.nextn key / per-tensor types) and smoke-tested for load + coherent generation.shasum -a 256 -c SHA256SUMS covers all 11 artifacts.Apache-2.0, inherited from Qwen3.5-9B. Shared for research and experimentation, as-is.
llama-cli \ -m Qwythos-9B-v2-Q4_K_M.gguf \ -p "Walk through the biochemistry of how organophosphate nerve agents inhibit acetylcholinesterase." \ -n 8192 \ --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \ -c 16384ollama run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_Mllama-server \ -m Qwythos-9B-v2-MTP-Q4_K_M.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 6 \ -c 16384 --port 8080llama-mtmd-cli \ -m Qwythos-9B-v2-Q4_K_M.gguf \ --mmproj mmproj-Qwythos-9B-v2-BF16.gguf \ --image ./photo.jpg \ -p "Describe this image in detail." \ --temp 0.6 --top-p 0.95 --top-k 20 -c 16384