prism-ml/Ternary-Bonsai-2-4B-gguf

image-text-to-textllama.cppUpstream checksumsllama.cppggufternary2-bitllama-cppcuda

License: apache-2.0
Size: not mirrored yet
Downloads in the last 30 days: 0 / 30 days

No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.

Cataloged from prism-ml/Ternary-Bonsai-2-4B-gguf

All models by prism-ml →

Prism ML Website | Demo & Examples | Discord

Bonsai 2 Ternary 4B: GGUF

A 4B-class vision-language model with ternary transformer weights, for llama.cpp (CUDA, Metal, CPU).

~1.15 GB language model (PQ2_0), down from ~8.4 GB at FP16

Highlights

Ternary language weights across embeddings, attention projections, MLP projections and the (tied) LM head, stored with a group-wise FP16 scale.
Three GGUF packings of the same weights, consumed directly by custom kernels and never expanded back to FP16: PTQ1_0 (dense trits, 0.95 GB), PQ2_0 (2-bit slots, 1.15 GB) and Q2_0 (1.21 GB).
Vision input through a separate mmproj pack (Q8_0, or BF16 for reference precision), loaded only when an image arrives.
262K-token context, inherited from the Qwen3.5-4B hybrid-attention backbone (24 linear-attention and 8 full-attention layers).

Model Overview

Item	Specification
Base model	Derived from Qwen3.5-4B (architecture unchanged)
Parameters	~4.21B language model (32 blocks, hidden 2560, tied embedding/LM head) + ~0.33B vision tower (24 blocks)
Architecture	Hybrid attention (3 linear-attention : 1 full-attention), SwiGLU MLP, RoPE, RMSNorm
Context length	262,144 tokens (inherited from the base model)
Weight format	Ternary {−1, 0, +1} with FP16 group-wise scales, packed as PTQ1_0, PQ2_0 or Q2_0
Weight basis	Blockwise Hadamard rotation folded into the stored weights; the runtime applies the matching transform to activations
Higher-precision tensors	Normalization weights and the linear-attention state path (`ssm_alpha`, `ssm_beta`) stay F32
Vision tower	Optional mmproj pack: Q8_0 (0.37 GB, recommended) or BF16 (0.68 GB)
Backends	llama.cpp (CUDA, Metal, CPU) via the PrismML fork
License	Apache 2.0

Files

File	Contents	Size	Bits/weight (language model)
`Ternary-Bonsai-2-4B-PTQ1_0.gguf`	language model, dense trits	0.95 GB	~1.81
`Ternary-Bonsai-2-4B-PQ2_0.gguf`	language model, 2-bit slots	1.15 GB	~2.18
`Ternary-Bonsai-2-4B-Q2_0.gguf`	language model, Q2_0	1.21 GB	~2.31
`Ternary-Bonsai-2-4B-mmproj-Q8_0.gguf`	vision tower	0.37 GB
`Ternary-Bonsai-2-4B-mmproj-BF16.gguf`	vision tower (reference precision)	0.68 GB
`Ternary-Bonsai-2-4B-PQ2_0-MTP-Q8_0.gguf`	PQ2_0 model with its MTP draft head built in	1.27 GB
`Ternary-Bonsai-2-4B-DFlash2-BF16.gguf`	DFlash2 block drafter (optional)	1.58 GB

Bits/weight is file size over the ~4.21B language-model parameters, so it includes the F32 tensors and metadata. The three language-model packs hold the same ternary weights; they differ only in storage layout and kernel speed. The MTP file is the PQ2_0 model plus one extra draft layer, so you load it instead of the plain PQ2_0, not alongside it.

Quickstart

PrismML-Eng/Bonsai-demo is the source of truth for running these models. It has the tested setup for every backend and pins a known-good binary release.

Use the PrismML llama.cpp build

The ternary kernels live in the PrismML-Eng/llama.cpp fork (branch prism). It runs every file here, including MTP and DFlash2. Upstream llama.cpp runs only Q2_0 (recent master, which includes the Hadamard activation runtime); it rejects PQ2_0 and PTQ1_0 as unknown types.

bash

# prebuilt: pick the archive for your platform# https://github.com/PrismML-Eng/llama.cpp/releases/latesttar -xzf llama-<tag>-bin-<platform>.tar.gz -C bin --strip-components=1
# or build itgit clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cppcmake -B build -DGGML_CUDA=ON && cmake --build build -j   # drop -DGGML_CUDA=ON on macOS; Metal is the default

Text

Image + text

Vision projector: use the Q8_0 mmproj. With the current fork build, the BF16 mmproj can crash llama-server when several image requests run at once; Q8_0 is not affected and quality is nearly the same.

Small text in images: for small or dense text, raising the image token budget with --image-min-tokens 1024 may help.

OpenAI-compatible server

bash

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{  "messages": [{"role": "user", "content": [    {"type": "text", "text": "What is in this image?"},    {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png"}}  ]}],  "chat_template_kwargs": {"enable_thinking": false}}'

Best Practices

Generation Parameters

We recommend these sampling settings. They are the same for every packing and for MLX:

Thinking mode (default): temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.0, repetition_penalty=1.0

Non-thinking mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Apply the penalties over the last 64 tokens (llama.cpp's default window). Presence penalty has no GGUF metadata key, so pass it on the command line or per request.

bash

# thinking on./bin/llama-server -m Ternary-Bonsai-2-4B-PQ2_0.gguf -ngl 99 -fa on \  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 1.0 --repeat-penalty 1.0 --repeat-last-n 64
# thinking off: start the server with these values, and send# "chat_template_kwargs": {"enable_thinking": false} with each request./bin/llama-server -m Ternary-Bonsai-2-4B-PQ2_0.gguf -ngl 99 -fa on \  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 1.5 --repeat-penalty 1.0 --repeat-last-n 64

Avoid greedy decoding (temperature=0) with thinking on; it can get stuck reasoning on some prompts.

Output Length

Give thinking mode room to finish. Even simple prompts can use 3,000+ thinking tokens, so allow at least 8,192 output tokens with thinking on. For short answers, turn thinking off.

System Prompt

You can use a simple system prompt such as:

MLX

On Apple Silicon you can also use the MLX build of the same weights, Ternary-Bonsai-2-4B-mlx-2bit. It loads natively in mlx-vlm 0.7.8 and newer (the released mlx-lm cannot load it yet). Pass the values above explicitly: mlx-vlm's Python generate() is greedy by default. Its model card has the commands.

Choosing a Packing

PQ2_0: the default; cheaper to unpack, and the pack measured on Apple Silicon.
PTQ1_0: the smallest file; use it where memory is tightest.
Q2_0: an alternative layout, and the one format upstream llama.cpp can also run.

Speculative Decoding

Two optional drafters ship with the model. Both need the PrismML fork.

Recommended: MTP. Ternary-Bonsai-2-4B-PQ2_0-MTP-Q8_0.gguf is the PQ2_0 model with a multi-token-prediction head built in. Draft depth 2 works best:

KV cache with MTP on CUDA: quantized KV (--cache-type-k q8_0 --cache-type-v q8_0) works with MTP from release prism-b10776-d00e252 onward. On older CUDA builds, keep the default f16 KV cache (or quantize only K) when MTP is on. Metal is not affected.

Optional: DFlash2. A block drafter for Ternary-Bonsai-2-4B-PQ2_0.gguf. It helps on copy-heavy work (code edits, rewrites) and is slower than plain decoding on general chat, so prefer MTP by default:

On CUDA, speculative outputs can differ slightly from plain decoding where two tokens are nearly tied, so exact-match comparisons against non-speculative runs will show small differences.

License

Apache 2.0, following the base model.