No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from AxionML/Gemma-4-12B-FP8
All models by AxionML →Developed by AxionML for open-source serving and deployment use cases. Part of AxionML's effort to provide ready-to-serve quantized models for the community.
This is an FP8-quantized version of google/gemma-4-12B-it (11.95B params), using per-block 128×128 weight-only FP8 (E4M3) with an FP8 KV-cache, MSE-calibrated. Activations are kept in BF16 — this is deliberate: gemma-4's attention activations carry large per-channel outliers, so quantizing activations (per-tensor W8A8) measurably degrades the model. Quantizing only the weights is lossless on GSM8K (0.9666 vs 0.9636 BF16) while halving the weight footprint (~24 GB → ~13 GB). Serves on Hopper (H100/H200) and Blackwell.
This model was quantized by applying per-block (128×128) FP8 (E4M3) to the weights of the linear operators within the transformer blocks. Activations are kept in BF16 (weight-only). The KV-cache is quantized to FP8 (E4M3). lm_head and the multimodal (vision/audio) embedders are kept in their original BF16 precision.
Requires the SGLang branch in SGLang support below (fp8_pb_wo block-FP8 support + Blackwell UE8M0 scale requant + transformers≥5.10 multimodal name handling for Gemma-4).
Multi-Token Prediction with the paired google/gemma-4-12B-it-assistant draft
works on this quantized target with the SGLang branch below. Use the Triton
attention backend and load the draft unquantized:
MTP is lossless on GSM8K (see Accuracy). Earlier SGLang mis-loaded
ModelOpt's attention-projection scales (self_attn.{k,v}_proj.{k,v}_scale) as
the RadixAttention KV-cache scales, which corrupted the spec-decode verify
forward on quantized targets (degenerate output) while BF16 targets were fine.
The branch fix leaves gemma-4's KV scales at their identity default (1.0) —
correct, because gemma-4 writes K/V to the cache after q/k-norm and RoPE, so
the projection-output scales are the wrong descale factor. (The related
trtllm_mha SWA-pool crash,
sgl-project/sglang#26957,
is already fixed on main.)
Sampling defaults for Gemma 4: temperature=1.0, top_p=0.95, top_k=64. Thinking mode is off by default; enable with extra_body={"chat_template_kwargs": {"enable_thinking": True}}.
Smoke test:
(--weight_calib_algorithm mse is a small local addition to ModelOpt's hf_ptq.py that overrides the qformat's weight calibration to MSE; fp8_pb_wo's stock algorithm is max.)
FP8 (E4M3 — 4 exponent bits, 3 mantissa bits, range ±448) stores each weight in a single byte. fp8_pb_wo ("per-block weight-only") quantizes weights in 2D 128×128 blocks, each with its own FP8 scale, and does not quantize activations — at runtime the block-scaled FP8 weights drive a block GEMM (DeepGEMM on Blackwell, with UE8M0 scale requant) against BF16 activations. Per-block scaling adapts to local weight magnitude far better than a single per-tensor scale, and MSE calibration sweeps each block's scale to minimize ‖W − dequant(quant(W))‖² instead of taking max-of-abs. The KV-cache is additionally stored in FP8 (E4M3) to halve KV memory.
Why weight-only on Gemma-4: standard per-tensor W8A8 FP8 (quantized activations, as NVIDIA ships for Llama/Nemotron) degrades sharply on gemma-4 — its attention residual stream has persistent per-channel activation outliers that a single per-tensor activation scale crushes, even within FP8's ±448 range. Leaving activations in BF16 (weight-only) sidesteps this entirely and is lossless.
A companion AxionML/Gemma-4-12B-NVFP4 ships a 4-bit variant following NVIDIA's dense-Gemma-4 recipe (nvidia/Gemma-4-31B-IT-NVFP4): NVFP4 (E2M1 + FP8 16-element micro-block scales) on the MLP/FFN, attention kept BF16, FP8 KV-cache, MSE-calibrated. NVFP4 requires Blackwell (native FP4 Tensor Cores); serve it with --quantization modelopt_fp4.
GSM8K (1319 questions, sgl-eval, greedy, served on SGLang):
MTP (greedy, exact verify) is lossless within GSM8K run-to-run noise — accuracy holds with and without speculative decoding.
Latency/throughput measured with NVIDIA AIPerf on the nvidia/SPEED-Bench qualitative split (all 11 domains, 880 prompts each issued once, shuffle / seed 42), greedy, output capped at 512 tokens, OpenAI chat + streaming, one Blackwell GPU, served on the SGLang branch below. Prompts are short (ISL ≈ 145, OSL ≈ 410 tokens). MTP uses the google/gemma-4-12B-it-assistant NEXTN draft.
Concurrency 1 — single-stream latency (the low-latency serving regime):
Concurrency 32 — throughput (saturated / compute-bound):
At saturation the GPU is compute-bound, so the fp8_pb_wo block-GEMM (with dequant) doesn't beat BF16 dense GEMM on aggregate throughput (0.90×), and MTP is roughly neutral (0.96×). Takeaway: FP8 — especially with MTP — pays off most in the low-concurrency / latency-bound regime; at saturation, throughput is comparable across formats.
Gemma 4 (including the encoder-free unified 12B) is supported on SGLang main. Serving this fp8_pb_wo checkpoint additionally needs the branch below, which adds: (1) fp8_pb_wo per-block weight-only FP8 to the ModelOpt FP8 path (the stock path is per-tensor only) plus Blackwell UE8M0 scale requant for the DeepGEMM block kernel; (2) remap of the embed_vision.* multimodal weight names emitted by a transformers≥5.10 ModelOpt re-export. It also fixes speculative decoding (NEXTN/MTP) on quantized targets: SGLang must not load ModelOpt's attention-projection scales (self_attn.{k,v}_proj.{k,v}_scale) as the RadixAttention KV-cache {k,v}_scale — gemma-4 caches K/V post-norm/post-RoPE, so those are the wrong descale factor and corrupt the spec verify forward; the KV scales correctly default to 1.0.
Branch: bzhng-development/sglang@gemma4-modelopt-ptq (off sgl-project/sglang main).
Serving needs the SGLang branch, so base it on a recent SGLang nightly image (lmsysorg/sglang:nightly-dev-YYYYMMDD-<hash>; cu13 variants exist for CUDA-13 hosts). The nightly already installs SGLang as an editable install rooted at /sgl-workspace/sglang, so the command below simply swaps that directory for the branch checkout — no reinstall needed — then pins the matching transformers, fetches the checkpoint, and starts the server, which will then be listening at http://0.0.0.0:30000 (change --port to use a different port):
--network=host publishes the server on the host's port 30000; alternatively drop it and use -p 30000:30000.launch_server line (HF_TOKEN is then required — the draft google/gemma-4-12B-it-assistant is gated).cd / matters: the image's default workdir is /sgl-workspace/sglang, so rm -rf-ing it from inside that directory makes git fail with "Unable to read current working directory."lmsysorg/sglang:nightly-dev-* tag also works — each ships the same editable /sgl-workspace/sglang layout this relies on.nvidia-smi inside the container reports a driver/library version mismatch. If so, add a mount matching your host driver (e.g. 580.82.07):
The base model was trained on data that may contain toxic language and societal biases. The quantized model inherits these limitations and may generate inaccurate, biased, or offensive content. Quantization can introduce additional deviations from the base model's behavior. Please refer to the original model card for full details.
google/gemma-4-12B-it is Google DeepMind's dense 11.95B-parameter Gemma 4 "Unified" (encoder-free) multimodal instruction-tuned model: text + image (+ audio) input, 256K context, hybrid sliding-window/global attention, configurable thinking mode, and native function calling. See the upstream card for full architecture, training data, evaluation, and responsible-AI details. This repository changes only the numeric precision of the weights — all capabilities, the chat template, and the tokenizer are inherited unchanged.
sglang serve --model-path AxionML/Gemma-4-12B-FP8 \ --quantization modelopt_fp8 \ --kv-cache-dtype fp8_e4m3 \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --mem-fraction-static 0.85 \ --host 0.0.0.0 --port 30000sglang serve --model-path AxionML/Gemma-4-12B-FP8 \ --quantization modelopt_fp8 \ --kv-cache-dtype fp8_e4m3 \ --attention-backend triton \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-12B-it-assistant \ --speculative-draft-model-quantization unquant \ --speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-eagle-topk 1 \ --reasoning-parser gemma4 --tool-call-parser gemma4 \ --mem-fraction-static 0.85 --host 0.0.0.0 --port 30000curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "default", "messages": [{"role": "user", "content": "What is C. elegans?"}], "temperature": 1.0, "top_p": 0.95, "top_k": 64, "max_tokens": 256}'python examples/llm_ptq/hf_ptq.py \ --pyt_ckpt_path google/gemma-4-12B-it \ --qformat fp8_pb_wo \ --weight_calib_algorithm mse \ --kv_cache_qformat fp8 \ --export_path ./gemma-4-12B-it-FP8 \ --trust_remote_code# Editable install of the branchgit clone https://github.com/bzhng-development/sglang.gitcd sglang && git checkout gemma4-modelopt-ptqpip install -e python
# transformers with Gemma 4 (encoder-free unified) supportpip install 'git+https://github.com/huggingface/transformers.git@1423d22f7a3b62e8c70ad67b58ec25cd9b675897'docker run --gpus all --shm-size=128g --network=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e HF_TOKEN=$HF_TOKEN \ lmsysorg/sglang:nightly-dev-20260604-14ed9b44 \ bash -lc ' cd / && rm -rf /sgl-workspace/sglang && git clone https://github.com/bzhng-development/sglang.git /sgl-workspace/sglang && cd /sgl-workspace/sglang && git checkout gemma4-modelopt-ptq && pip install "git+https://github.com/huggingface/transformers.git@1423d22f7a3b62e8c70ad67b58ec25cd9b675897" && python -m sglang.launch_server \ --model-path AxionML/Gemma-4-12B-FP8 \ --quantization modelopt_fp8 \ --kv-cache-dtype fp8_e4m3 \ --reasoning-parser gemma4 --tool-call-parser gemma4 \ --mem-fraction-static 0.85 \ --host 0.0.0.0 --port 30000 '-v /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.580.82.07:/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1:rogit clone https://github.com/NVIDIA/TensorRT-Model-Optimizer.gitcd TensorRT-Model-Optimizer && pip install -e ".[hf]" # commit f9423c0d3