Mirrored from cyjin-yl/Qwen3.8-27B-Uncensored-Cyber-agentic-imatrix-GGUF
All models by cyjin-yl →GGUF quantizations of philbert440/Qwen3.8-27B-Uncensored-Cyber, calibrated on an importance matrix built from real agentic-coding traffic rather than generic English prose.
All model weights and behaviour are philbert440's work. This repository contributes only the quantization and the calibration methodology — no fine-tuning, no merging, no behavioural change of any kind was performed here.
Inherited from upstream, and stated plainly so nobody is surprised: this is a de-refusal ("abliterated") model specialized for the cyber/offensive-security domain. The vision tower and the MTP speculative-decoding head are preserved upstream, which is what makes the multimodal path and the MTP graft work at all. Read the base model's card before using it — it documents the recipe and the evaluation.
Status (2026-08-20): the imatrix and both IQ4_XS artifacts are uploaded. Sizes and SHA-256 values below are the Hugging Face LFS objects, checked against the local production artifact for the MTP build.
The two model files have identical IQ4_XS/imatrix text weights. The plus-mtp file adds
the MTP tensors from the matching upstream model; use it only with a runtime that
recognizes that graft. For a conventional GGUF loader, use the non-MTP file unless its
MTP support has been verified. The Q5_K_M comparison below was measured locally and is
not published in this repository.
The usual llama.cpp calibration sets (wiki.train.raw, groups_merged.txt) are generic
prose. The tokens that actually break under quantization in agentic use are a different
population:
bash, read, web_search)<|im_start|>, <tool_call>)Those barely occur in generic prose, so a prose-derived imatrix treats their channels as
unimportant and the quantizer spends its error budget there first. The concrete failure
this targets: an agent writing /home/eze/Documents/PotouI for
/home/ezra/Documents/Proto-UI, then reading its own mangled output back and concluding
its context is inconsistent.
Extracted from real agent session logs (an agent CLI driving coding tasks against this model) — not synthetic prompts:
The corpus itself is not published. It is private working data — real repository
paths, real shell commands, real conversations. The resulting imatrix-agentic-v2.gguf is
published so the method is reproducible; rebuild your own corpus from your own traffic to
reproduce it end to end.
Three choices worth explaining:
--parse-special is not optional here. The corpus is rendered with the model's own
chat-template markers. Without that flag llama.cpp tokenizes <|im_start|> as literal
text — <, |, im, _start, … — so the special tokens never enter the statistics at
all, which defeats the point of building this corpus. Measured effect on the same corpus
family: 0.342 → 0.299 tokens per byte (−12.7%), consistent with those markers collapsing
into single tokens.
Context stayed at 512, deliberately. The intuitive move is to match the 262 K context this model is served at. Community testing says the opposite: for a fixed token budget a small context usually beats 4096, because it yields more and more-diverse samples and a better-conditioned estimate. 512 is also llama.cpp's long-standing default.
--output-tensor-type q8_0 --token-embedding-type q8_0. Output-layer quantization
error turns directly into picking the wrong token, which is precisely the verbatim-copy
failure above. Costs roughly 1.5 GB.
--allow-requantize). Q8_0 is
near-lossless and the error is dominated by the Q5/IQ4 step, but this is still a second
quantization.Built while making this model usable for agentic work on a single V100 32 GB (SM70) —
FP16-only tensor cores, no INT8/FP8/FP4 TC, no cp.async. On that hardware decode is
memory-bandwidth bound, so IQ4_XS (~4.25 bpw) reads about 26% fewer bytes per token than
Q5_K_M (~5.33 bpw) and frees roughly 4 GB for KV cache — which is why it is the primary
target here rather than the largest quant that fits.
The production configuration used for the measurements below is FastLLM on one V100:
IQ4_XS-imatrix-fromq8-plus-mtp, turbo3 KV cache, batch 4, and a 262,144-token
limit. Multimodal use also requires the matching Qwen3.8 vision projector; it is a
separate artifact and is not embedded in either text-model GGUF here.
Quantized from philbert440's own Qwen3.8-27B-Uncensored-Cyber-Q8_0.gguf release.
llama-imatrix and
llama-quantize. The importance-matrix approach itself comes from
PR #4861; the GGUF imatrix format
from PR #9400.apache-2.0, inherited from the base model. These are quantizations of philbert440's
weights; the upstream terms govern.
留出集与校准语料的 512 字节窗口重叠率 0.00% / 0.11%,是干净评测集。
三组用同一份留出集、同一个 --chunks 60。
不要按这两个置信区间去比。 两组评的是同一批分块,误差高度相关, 按独立区间读会严重低估判别力。做配对比较(逐块还原 NLL 后对差值做检验):
也就是说:从 5.33 bpw 降到 4.25 bpw 的代价是 0.42%,差异真实、方向一致, 但幅度很小。配对把不确定度从 ±1.74% 压到 ±0.187%,约 9 倍。
"逐字抄写"测的是模型把上下文里的路径、UUID、commit 哈希原样复制的能力。 它比困惑度更贴近 agent 的真实失败模式:抄歪一个字符命令就失败,而困惑度 是全词表平均,对这种尾部行为几乎不敏感。
早期把不同上下文长度的请求直接比较,一度误判 IQ4_XS 比 Q5_K_M 慢。控制成 相同 token 形状后,8 组端到端配对样本全部同向:IQ4_XS 平均为 Q5_K_M 的 1.20x。这是单卡 V100 + FastLLM 的部署结果,不是跨推理栈的通用保证。
当前生产仍有明确的算子余量:decode 的主路径是通用 mmvq,而 SM70 IQ4_XS
MMQ 只覆盖较大的 n;因此 1.20x 主要来自更低的权重带宽和当前融合路径,
不代表专用 decode kernel 已经用满硬件。
# importance matrix, computed on the Q8_0 releasellama-imatrix -m Qwen3.8-27B-Uncensored-Cyber-Q8_0.gguf \ -f calib-agentic.txt \ -o imatrix-agentic-v2.gguf \ -ngl 99 -c 512 --parse-special
# quantization (IQ4_XS shown; Q5_K_M identical apart from the type)llama-quantize --allow-requantize --imatrix imatrix-agentic-v2.gguf \ --output-tensor-type q8_0 --token-embedding-type q8_0 \ Qwen3.8-27B-Uncensored-Cyber-Q8_0.gguf out.gguf IQ4_XSQwen3.8-27B — Qwen team └─ philbert440/Qwen3.8-27B-Uncensored-Aggressive α=1.15, recipe-v2 refusal ablation └─ philbert440/Qwen3.8-27B-Uncensored-Cyber residual-cyber peel, β=1.0, apply_from=4 └─ this repository quantization onlyIQ4_XS vs Q5_K_M ΔPPL = +0.417% ± 0.187% t = 2.22 60 块中 39 块更差