No official torrent yet โ requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from yuxinlu1/Mellum2-12B-A2.5B-Claude-4.6-4.8-Opus-Thinking-GGUF
All models by yuxinlu1 โ12B total params, only 2.5B active per token. This is a Mixture-of-Experts model, so it runs like a ~2.5B model but thinks like a 12B one. ๐ Built on JetBrains' Mellum 2 (a from-scratch software-engineering model) and tuned on Claude Opus 4.6 / 4.7 / 4.8 reasoning traces โ it reasons step-by-step in
<think>blocks, then answers. ๐ง ๐ป All local, all yours, no API, no cloud. And it's seriously fast.
~440 tokens/sec on a single RTX 5090 at Q4_K_M (--n-gpu-layers 99 -fa on) โ and generation quality
holds up: correct, coherent code and clean step-by-step reasoning. ๐ฏ
You get big-model answers at small-model speed. Why so quick? It's a Mixture-of-Experts (only 2.5B of the 12B params fire per token) with a compact 98K vocab โ so it generates several times faster than a dense model its size, with no draft / speculative model needed. ๐
๐ก It's a Mixture-of-Experts: all 64 experts live on disk/VRAM (so size is for the whole 12B), but only 8 fire per token โ that's why it's so quick.
Mellum2 has a tiny KV cache (GQA with just 4 KV heads, and sliding-window attention on 3 of every 4 layers) โ so context is rarely the limiter. Pick the quant that fits your VRAM and you'll have plenty of room for long context (max is 131K). Rough numbers ๐ค (weights + ~2 GB overhead):
๐ก Apple Silicon / iGPUs with unified memory count too โ same idea, just slower than a dGPU. ๐ก Tight on room? Drop a quant or use a
q4_0KV cache for even more context.
โฆ-Q4_K_M.gguf) and llama-server from llama.cpp.
โ ๏ธ Needs a recent llama.cpp that supports the
mellum2architecture (a mid-2026 build or newer). Older builds fail withunknown architecture: 'mellum2'.
.bat shown โ tweak --port, --ctx-size to taste):http://localhost:18080 and chat. ๐ (Tip: bump --ctx-size โ the KV cache is small, so go big.)Works in LM Studio, Jan, Ollama, etc. โ just import the GGUF, pick your quant, go. ๐พ
(Make sure the app ships a recent llama.cpp that knows mellum2.)
This model thinks natively in <think> โฆ </think> blocks. The chat template handles it automatically;
the --reasoning-format deepseek flag tells llama.cpp to surface the reasoning cleanly.
Recommended sampling: temp 0.6, top_p 0.95, top_k 20 (JetBrains' official settings for the Thinking model).
This repo ships GGUF (for llama.cpp). To run in raw transformers you need non-GGUF weights โ point
midat the original JetBrains checkpoint (or your own merged fp16). Needstransformers >= 5.8(themellumarchitecture is built in โ notrust_remote_code). It's a plain text CausalLM (not multimodal).
JetBrains' Mellum 2 uses Multi-Token Prediction as a training-time objective only โ it isn't exported to the released weights, so there's no draft to ship. You don't need one: with just 2.5B active params and a compact vocab, it already generates extremely fast (~440 tok/s on a 5090 at Q4_K_M). ๐๏ธ
JetBrains/Mellum2-12B-A2.5B-Thinking
โ a from-scratch, software-engineering-focused Mixture-of-Experts model (12.15B total / 2.5B active,
28 layers, 64 experts with 8 active, 131K context). JetBrains already did SFT + RLVR on it; this is a light
extra LoRA distillation pass on top.JetBrains/Mellum2-12B-A2.5B-Thinking,
released under Apache-2.0.angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k,
augmented with additional Opus 4.8-generated reasoning samples I curated and mixed in.@echo offcd /d C:\llama.cppllama-server.exe ^ -m C:\models\mellum2-claude-Q4_K_M.gguf ^ --ctx-size 16384 ^ --n-gpu-layers 99 ^ --no-mmap ^ -fa on ^ --jinja --reasoning-format deepseek ^ --temp 0.6 --top-p 0.95 --top-k 20 ^ --host 0.0.0.0 --port 18080pauseimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer
mid = "JetBrains/Mellum2-12B-A2.5B-Thinking" # GGUF won't load here โ use non-GGUF weightstok = AutoTokenizer.from_pretrained(mid)model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.bfloat16, device_map="auto", attn_implementation="sdpa")
msgs = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]inputs = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True, enable_thinking=True)inputs = inputs.to(model.device)out = model.generate(inputs, max_new_tokens=512, temperature=0.6, top_p=0.95, top_k=20)print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=False))