Mirrored from autotrust/GEV-26B-Decide-NVFP4
All models by autotrust →This repository is autotrust/GEV-26B-Decide with its 3,840 routed expert MLPs (30 layers × 128 experts) quantized to NVFP4. Everything else is the bf16 model unchanged: attention, the dense MLP of every layer, the routers, the vision tower, the lm_head, the System 1 adapter, the decision head and the temperatures. The rest of this card is the card of GEV-26B-Decide; its benchmark tables are the bf16 results, and the NVFP4 comparison is below.
GPQA Diamond and HLE are the Decision Index items; the bf16 row is the model's transformers engine, the NVFP4 row is
vLLM with this checkpoint. The differences are within run-to-run noise (198 questions: about ±7 points).
What is quantized and how. NVFP4 is NVIDIA's 4-bit floating-point format: FP4 E2M1 values in groups of 16 with an
FP8 E4M3 scale per group and an FP32 scale per tensor. Per expert, gate, up and down projections are quantized (gate
and up share one tensor scale, as vLLM fuses them); activations of the experts use static scales calibrated on 3,072
System 1 prompts from GEV's own calibration splits. The checkpoint uses the ModelOpt NVFP4 layout and the same
exclusion list as nvidia/Gemma-4-26B-A4B-NVFP4 (quant_method: modelopt), so vLLM loads it without any flag.
Why only the experts. The routed experts hold most of the weights (about 22 GB of FP4-able bf16 in 24 B parameters),
and the System 1 adapter does not touch them, so adapter_vllm/ is the same as for the bf16 model and System 1 needs no
retraining. Attention, the dense MLP and the router carry the LoRA or are small; quantizing them saves little memory.
serve.sh and the API are exactly those of GEV-26B-Decide (see Quick start); vLLM reads the
quantization from config.json / hf_quant_config.json and logs modelopt_fp4 and the selected NVFP4 MoE kernel.
--gpu-memory-utilization
and MAX_MODEL_LEN to fit the card, e.g. MAX_MODEL_LEN=32768.patches/vllm-gemma4-lm-head-lora.patch (LoRA on
Gemma-4's tied lm_head); tested with a vLLM development build from September 2026.<bos>; serve_decide.py adds it. If you call
/v1/completions yourself, start the prompt with <bos> and pass top_k: 0 and top_p: 1.0.transformers + peft path below loads bf16 weights; use the bf16 repository
autotrust/GEV-26B-Decide for it.How the score was computed (our scoring with the kit's score --edition 0.2.1, not a board entry):
autotrust/jev-decision-index-results
(runs/jev-gemma4-26b-a4b, the weights' previous name). Thinking was tried on five of their benchmarks and is not used
there: it adds little to classification, retrieval and tool selection (see
Outside Knowledge & Reasoning).Details: reports/decision_index_adaptive.json, reports/adaptive_latency_summary.json.
The fastest vision model of the family. Every step below is one System 1 decision (thinking off): a screenshot or a camera image in, a probability for every action out, in a single forward pass on one B200.
Computer use: screenshot → which element to click. A real browser (headless Chromium). Every clickable element gets a numbered box; System 1 picks the next click (or "the task is complete"), the browser clicks it, and the loop repeats.
95% of 60 random multi-step tasks completed (shop, settings, mail; 3–7 clicks each) in about 85 ms per click: the same success rate as JEV-27B-VL, 3× faster. The colour swatches carry no text, so that click is decided from the screenshot alone.
Robot arm: pick and place from a camera image. At every step System 1 looks at the top camera image and answers two questions: is the target left or right of the gripper, and above or below it? The arm moves accordingly and halves its step whenever an answer flips. It grasps the cube, carries it and drops it in the tray (MuJoCo simulation).
61 ms per decision, so a whole pick and place takes 4–8 seconds of model time. It completed 40% of 20 random scenes: close to the target its left/right answers are less precise than JEV-27B-VL's, so more grasps miss. Once grasped, 8 of 9 cubes ended in the tray.
Same scenes and tasks for every model in the family:
Demo code: JEV-9B vl/demos/ (set JEV_URL to this
server). Per-episode results: reports/demos/ [blocked].
GEV-26B-Decide answers typed questions with a calibrated probability for every option; with thinking switched on, it thinks only when it needs to. System 1 decides in one forward pass (about 45 ms). When its leading option is uncertain, System 2 (the same backbone in Gemma-4 thinking mode) reasons over the question, and the reasoning is folded into the final probabilities. One set of weights, one vLLM engine, for text and images.
GEV-26B-Decide was previously published as autotrust/JEV-Gemma4-26B-A4B; the weights are the same.
Two models, two organisations. TypeSafe Jev 1.13 is the hosted, closed model made by TypeSafe AI. autotrust/GEV-26B-Decide is an independent open-weights model built by AutoTrust AI; it is not affiliated with, endorsed by, or a product of TypeSafe AI.
Each puzzle has one correct answer. System 1 answers in one pass; with thinking: "auto", System 2 thinks when System 1
is below 0.8 confidence, and its answer is folded into the probabilities. The videos show puzzles that System 1 got wrong;
the tables below count all puzzles.
Minesweeper. Which hidden cell is certainly safe? System 1 is at chance (21.0 % against 25 %); with thinking, 86.0 %. These thoughts usually reach the 8,192-token budget, and the answer read at that point is still right most of the time.
Connect Four. Which column wins now, or stops the opponent from winning next move? 54.0 % → 99.5 %.
Wordle. Which word still fits all the colour feedback? 52.0 % → 100 %.
Sudoku. Which digit belongs in the highlighted cell? 76.0 % → 99.5 %.
One-move puzzles, 200 generated puzzles per game (threshold 0.8, budget 8,192 thinking tokens; text input, chess with the board image as well):
Whole games and perception, where thinking helps little or not at all:
Thinking pays off when the answer can be checked step by step against explicit rules: logic puzzles, tactics,
constraints, arithmetic. It does not help perception (sketches), reflexes (Flappy Bird) or long-horizon play (2048, full
Connect Four games), and every thought costs seconds. Details: reports/thinking_games.json.
think_budget caps the thinking tokens (default: no cap beyond the context window; the evaluations below
used 8,192).The threshold and the mix were chosen on 1,754 questions from six public sets that are not part of the Decision Index (test or validation splits, 300 random questions each; AQuA-RAT has 254):
Accuracy in %. Calibration is unchanged: ECE 0.035 for System 1 and 0.035 for adaptive. The adaptive mode reaches 96 % of the always-think gain while thinking on 48 % of the questions. Thinking length: median 2,282 tokens, 90th percentile 7,573; 91.6 % of the thoughts finish within the 8,192-token budget.
All ten benchmarks of the area (33,856 scored requests) ran through the server (POST /v1/decide, thinking: "auto",
threshold 0.8, budget 8,192 thinking tokens) and were scored with the Decision Index kit:
Accuracy in %; "chance" is the kit's random baseline, and the Decision Index skill rescales accuracy so that chance is 0 (below chance counts as 0). The Knowledge & Reasoning area skill rises from 0.429 to 0.602.
Adaptive thinking with the same settings on five benchmarks of the other areas (run stopped after these):
Small gains on tool calls and intents, none on retrieval, and a loss on BANKING77, whose training split System 1 was trained on: the untrained System 2 often overrules a correct System 1 with an over-confident wrong answer. Use thinking for reasoning questions; keep it off for classification, retrieval and tool routing.
Thinking costs time. Over the area's ten benchmarks, 66 % of the requests thought at least once. Thinking length:
median 8,192 tokens (the budget) on GPQA, HLE and ChessBench, about 5,400 on MMLU-Pro, 3,300 on BBH and 900 on GSM8K.
Estimated single-request latency on one idle B200 (System 1 ≈ 45 ms, thinking ≈ 250 tokens per second): about 0.05 s
without thinking and up to about 33 s with an 8,192-token thought; median 13.4 s over the area (90th percentile 33 s).
With speculative decoding (below) thinking runs at about 438 tokens per second: median 7.7 s, 90th percentile 19 s.
Lower threshold or think_budget to trade accuracy for speed; thinking: "off" keeps every decision in one pass.
MMLU-Pro and BBH (except six questions) ran with speculative decoding, which does not change the output distribution.
Six BBH questions with 18 options hit a vLLM error in the speculative-decoding read-out path and were answered by the same
server without speculative decoding; the current serve_decide.py reads answers through a path that works with
speculative decoding.
The checkpoint contains Gemma-4's vision encoder (no audio encoder), so System 1 and System 2 both accept images. The decision head was trained on text; decisions over images are zero-shot.
For reference, autotrust/JEV-27B-VL scores 78.3 % on VL-RewardBench with the same protocol.
The backbone's native context is 262,144 tokens (256K). We tested decisions that hinge on a single sentence placed at a random depth in long real text (concatenated PubMedQA abstracts): a yes/no question and a 16-option question, 10 of each per length, on one B200 with vLLM, one request at a time.
Lengths above 128K have not been tested yet.
choice takes 2–256 options. Up to 16 are read in one pass with the trained labels A–P. More options are read in groups
of at most 16 (in parallel), then a final of 16; every option is read and none is pruned (strategy: "tournament", the
default). Zero-shot intent classification, all options offered at once, 400 test utterances per row:
The 255-option sets merge three catalogues with overlapping intents, so part of the drop comes from near-duplicate labels.
A single pass with labels beyond P (strategy: "single") is about 3× faster but less accurate here (CLINC150: 88.5 %
against 95.2 %), so it is not the default. The BANKING77 and CLINC150 training splits are part of the training data (see
below).
serve.sh runs serve_decide.py: the standard vLLM OpenAI server (same flags as vllm serve) with a POST /v1/decide
route. It loads the backbone once: plain requests are System 2, and requests for the LoRA module jev-decision
(adapter_vllm/: backbone LoRA + the decision head as an lm_head LoRA) are System 1. It needs a vLLM build with
Gemma-4 support plus patches/vllm-gemma4-lm-head-lora.patch (LoRA on Gemma-4's tied lm_head, vocabulary 262,144);
tested with a vLLM development build from September 2026.
POST /v1/decideThe response has options, probabilities, choice, choice_index, usage and, when thinking was requested,
thinking: {"used": true, "think_tokens": …, "think_seconds": …, "finished_within_budget": …}. GET /v1/decide/info
lists the defaults.
transformers engine below to a mean largest probability difference of 0.015 (300 held-out decisions).Faster thinking with speculative decoding. MTP=1 bash GEV-26B-Decide/serve.sh adds Google's 0.9 GB draft model for
this backbone (--speculative-config '{"model": "google/gemma-4-26B-A4B-it-assistant", "num_speculative_tokens": 4}').
On one B200 it speeds up System 2 about 1.8–1.9×: 247 → 438 tokens per second for a single request, 7,072 → 13,505 tokens
per second at 128 concurrent requests (mean acceptance length 3.5–3.7 of 4). System 1 decisions are unchanged, but System 1
throughput at high concurrency drops (257 → 140 decisions per second with 64 clients); use it when you mostly think.
serve_decide.py reads answers with the token restriction that works under speculative decoding, which needs
--max-logprobs 256 (set in serve.sh).
If you call /v1/completions for System 1 yourself, pass top_k: 0 and top_p: 1.0: the model's generation config sets
top_k=64 and top_p=0.95, which vLLM applies as request defaults and which would truncate the returned probabilities.
This path needs the bf16 weights: snapshot_download("autotrust/GEV-26B-Decide") (the bf16 repository).
This path reads up to 16 options per pass; for more, use the server (or read groups of 16 and a final, as above).
bare-v1, prefixed with <bos>: [kind] … [state] … [question] … [options] A) … [decision]:Use calibration_gold.json when you gate automatic actions on confidence.
System 1 was trained on teacher distributions and ground-truth decision data. The ground-truth data includes the public training splits of some datasets whose test splits the Decision Index uses (among them BANKING77 and CLINC150); no test split of any benchmark was used, and suite items were excluded before training. The list has been provided to the Decision Index maintainers. MMMU / MMMU-Pro are not valid evaluations for this model. The adaptive-thinking settings were chosen on data outside the Decision Index suite.
noul and score accept only their canonical options.Apache-2.0 for the adapter, head and calibration files; base model under the Gemma 4 terms (https://ai.google.dev/gemma/docs/gemma_4_license). Not affiliated with TypeSafe AI.
hf download autotrust/GEV-26B-Decide-NVFP4 --local-dir GEV-26B-Decide-NVFP4MODEL_DIR=GEV-26B-Decide-NVFP4 bash GEV-26B-Decide-NVFP4/serve.sh # vLLM on :8000hf download autotrust/GEV-26B-Decide-NVFP4 --local-dir GEV-26B-Decide-NVFP4MODEL_DIR=GEV-26B-Decide-NVFP4 bash GEV-26B-Decide-NVFP4/serve.sh # vLLM on :8000; one GPU with 24 GB or morecurl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{ "kind": "choice", "state": "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.", "question": "How much does the ball cost?", "options": ["$0.10", "$0.05", "$1.00", "$0.55"], "thinking": "auto"}'import requests
def decide(kind, state, question, options=None, thinking="auto"): body = {"kind": kind, "state": state, "question": question, "thinking": thinking, **({"options": options} if options else {})} r = requests.post("http://localhost:8000/v1/decide", json=body).json() return dict(zip(r["options"], r["probabilities"])), r.get("thinking", {}).get("used")
decide("noul", "John was born on 29 February 1996.", "Was John's 7th birthday celebrated on a 29 February?")requests.post("http://localhost:8000/v1/chat/completions", json={ "model": "autotrust/GEV-26B-Decide", "messages": [{"role": "user", "content": "In one sentence, what is safety stock?"}], "max_tokens": 200, "chat_template_kwargs": {"enable_thinking": False}})import json, torchfrom huggingface_hub import snapshot_downloadfrom peft import PeftModelfrom safetensors.torch import load_filefrom transformers import AutoTokenizer, Gemma4ForConditionalGeneration
d = snapshot_download("autotrust/GEV-26B-Decide")tok = AutoTokenizer.from_pretrained(d)base = Gemma4ForConditionalGeneration.from_pretrained(d, dtype=torch.bfloat16, device_map="cuda")# System 2: `base` is gemma-4-26B-A4B-it unchanged; use base.generate(...) (text or images).
# System 1: adapter merged in memory + headm = PeftModel.from_pretrained(base, f"{d}/adapter").merge_and_unload().eval()backbone = m.modeljc, T = json.load(open(f"{d}/judge_config.json")), json.load(open(f"{d}/calibration.json"))["per_kind"]head = load_file(f"{d}/head.safetensors"); W, b = head["proj.weight"].cuda(), head["proj.bias"].cuda()
@torch.no_grad()def decide(kind, state, question, options): lines = options if kind != "choice" else [f"{'ABCDEFGHIJKLMNOP'[i]}) {o}" for i, o in enumerate(options)] text = f"[kind] {kind}\n[state] {state}\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:" ids = torch.tensor([[tok.bos_token_id] + tok.encode(text, add_special_tokens=False)], device="cuda") h = backbone(input_ids=ids, use_cache=False).last_hidden_state[0, -1].float() z = 30.0 * torch.tanh((W @ h + b) / 30.0) s, _ = jc["slots"]["ranges"][kind] return dict(zip(options, torch.softmax(z[s:s + len(options)] / T[kind], 0).tolist()))
print(decide("noul", "Customer says the parcel arrived damaged and wants their money back.", "Is the customer asking for a refund?", ["false", "true"]))model-*.safetensors · config.json · hf_quant_config.json · processor_config.json · tokenizer* · chat_template.jinja · generation_config.json google/gemma-4-26B-A4B-it; routed experts NVFP4 (ModelOpt layout), everything else bf16 unchangednvfp4_activation_amax.json calibrated activation ranges of the experts (per layer)adapter/ System 1 LoRA (peft), for the transformers pathhead.safetensors 24-slot decision head (fp32): proj.weight [24, 2816], proj.bias [24]judge_config.json slot layout, verbalizer ids, softcap, read-outcalibration.json per-kind temperatures (default)calibration_gold.json per-kind temperatures calibrated against ground-truth answersadapter_vllm/ System 1 for vLLM: backbone LoRA + the head as an lm_head LoRA, plus decision_head.jsonserve_decide.py · serve.sh vLLM server with POST /v1/decide (System 1, adaptive thinking) next to the OpenAI endpointspatches/ vLLM patch: LoRA on Gemma-4's tied lm_headreports/ adaptive thinking: validation summary, Decision Index recomputation, latency summary, other-area sample, gamesreports/demos/ computer use and robot arm: per-episode resultsvideos/ adaptive-thinking videos (Minesweeper, Connect Four, Wordle, Sudoku); computer use and robot arm