autotrust/GLM5.3-Flash-E192-DGX-Spark
image-text-to-textvllmVerified checksumsautotrustglm5_nextmoenvfp4modeloptvllm
- License
- mit
- Size
- 130 GB
- Revision
- main
Mirrored from autotrust/GLM5.3-Flash-E192-DGX-Spark
All models by autotrust βGLM5.3-Flash-E192-DGX-Spark
π΄ Live: AutoTrust/GuruSearch recommendation & search demo β news.guru.so
[!IMPORTANT] Try AutoTrust/GuruSearch live (new, 11 October 2026). A recommendation and search demo in which every ranking is a calibrated System 1 decision, made in about half a second:
- Recommendation: the latest headlines from 6 news feeds, ranked by importance.
- Search: results from Google News, Bing News and Yahoo News re-ranked by relevance, side by side with the search engines' own order, plus a short answer with citations.
autotrust/GLM5.3-Flash-E192-DGX-Spark is a compact build of zai-org/GLM-5.3-Flash for desktop Blackwell systems like NVIDIA DGX Spark. It's an unofficial derivative.
It keeps 192 of the 288 routed experts in each layer by Neural Architecture Search (NAS), uses NVFP4 for the experts and still activates 18 B parameters per token. The weights take 124 GiB, the smallest in the family: two DGX Sparks connected by ConnectX-7 get about 62 GiB of weights per node and the most KV room of any build, and a single 180 GB Blackwell GPU (B200/GB200) runs it with 32 GiB of KV cache. The KV cache uses FP8.
The repo includes a ready-to-use NVFP4 MTP draft in mtp-nvfp4/ [blocked] (6.95 GB) for speculative decoding. On a single B200 it gives 1.80Γ single-stream decode speed, and output quality is unchanged.
The model keeps the full 154,880-token vocabulary and has the vision tower intact.
Benchmarks
Measured on one B200 with vLLM. Sampling follows the base model's recipe (temperature=1.0, top_p=0.95); HumanEval uses greedy decoding. reasoning_effort is the GLM-5.3-Flash chat-template thinking budget (low / high / max). The token budget is 65,536 (16,384 for MMMU).
Scoring is strict: a response that runs out of tokens before giving a final answer counts as wrong. All numbers are single runs. MoE decoding in vLLM isn't bit-deterministic, so treat Β±2β3 points as noise. E224 and E256 numbers come from the same harness on the same machine.
β‘ About the GPQA-Diamond numbers: the
effort=lowrow is a quick self-test with a 65,536-token budget, run to compare builds against each other. Atlowthe model thinks briefly and some answers are cut off before they finish. Theeffort=maxrow uses the GLM-5.3-Flash evaluation settings (temperature=1.0,top_p=0.95, up to 163,840 generated tokens), served with the NVFP4 MTP draft (speculative decoding is lossless). Atmax, 8 of the 198 answers still hit the 163,840-token budget before giving a final answer and count as wrong.
Tool use: BFCL v4 (function calling, AST match)
BFCL was run with bfcl-eval v4 against the local OpenAI-compatible endpoint (--tool-call-parser glm47, template-default effort, --num-threads 32).
Throughput and MTP speculative decoding (single B200)
reasoning_effort=low, 1,024 output tokens, short prompts, 8 requests per level, served from this repository as uploaded (--max-model-len 66560 --gpu-memory-utilization 0.93):
Text, Chinese, tool-calling and image smoke tests pass with MTP on. Speculative decoding is lossless: the target model verifies every drafted token, so the draft can only change speed, not output quality. With MTP on, the KV pool on one B200 drops from 2.6 M to 1.65 M tokens (25 concurrent requests at 66 K), so turn it off for very high-concurrency batch serving.
The mtp-nvfp4/ draft
- It is the original GLM-5.3-Flash MTP layer (one nextn layer, all 288 routed experts). Only the storage precision of its MoE weights changed.
- It is exported with NVIDIA ModelOpt (
NVFP4QTensor.quantize), the same format as the E224 repo'smtp-nvfp4/: the 288 routed experts and the shared expert are NVFP4 (e2m1 weights packed two per byte, E4M3 block scales over 16-element groups). Attention, the DSA indexer, the router,eh_proj, norms, embeddings andshared_headstay BF16. - One file, 6.95 GB, vs 17.4 GB for the BF16 draft.
- The same draft works with E192, E224 and E256: the MTP layer doesn't depend on how many experts the main model keeps.
- To regenerate it from the BF16 MTP layer (the
mtp/folder of E224), runscripts/export_mtp_nvfp4.py[blocked] (needsnvidia-modelopt):
Fixed 2026-10-10. The first upload of
mtp-nvfp4/(two shards) kept the shared expert in BF16, but its quantization manifest only excluded it under a module name that vLLM 0.31 doesn't use. On vLLM 0.31 the draft then failed to load with a shape assertion. It is now replaced by the ModelOpt export above. If you downloaded the old two-shard version, re-downloadmtp-nvfp4/.
Deployment
Requirements
- vLLM with GLM-5.3-Flash support. Validated on the
ZJY0516/vllm@glm-releasebranch (vllm-project/vllm#53906) at commit7e2d791plus8f8cc41("Make GLM-5.3 kpool metadata graph-safe"). Without that fix, full CUDA graphs can crash under concurrency. - For MTP: apply
scripts/vllm_glm5next_standalone_mtp_draft.patch[blocked] tovllm/models/glm5next/nvidia/mtp.py. The draft is a standalone directory with its own config (288 experts,num_nextn_predict_layers=1), and the patch makes vLLM build the MTP layer from the draft's config instead of the target's. transformers >= 5.16.1- Set
VLLM_USE_DEEP_GEMM=0.
Single B200 / GB200 (validated)
This loads 123.7 GiB of weights and leaves 31.8 GiB of FP8 KV cache: 2.6 M tokens, or 39 concurrent requests at 66 K. For reasoning_effort=max, use --max-model-len 172032.
With the NVFP4 MTP draft (interactive use), add:
Draft and target together load 127.9 GiB and leave a 1.65 M-token KV pool at 66 K context.
2Γ DGX Spark
The E224 two-Spark runbook (network setup, NCCL/CX7, swap, launch scripts) applies unchanged. Replace the model path and served name. At TP=2 this model puts about 62 GiB of weights on each node (E224: 71 GiB), so the same --gpu-memory-utilization leaves roughly 9 GiB more KV cache per node than E224. That's enough for reasoning_effort=max at 160 K+ context. The NVFP4 MTP draft adds ~3.5 GiB per node and is the recommended draft on Spark.
β οΈ We haven't run this model on DGX Spark hardware. The per-node figure above is estimated from the measured weight footprint. Reports from Spark owners are very welcome in the Community tab.
Request format
- Thinking is always on and comes back in the
reasoning/reasoning_contentfield. reasoning_effort:low,highormax(the default). Pass it as the top-level OpenAI field or viachat_template_kwargs.- Recommended sampling:
temperature=1.0, top_p=0.95. Greedy decoding can make long thinking loop on hard prompts. - Use
lowfor chat, Q&A, tool calls, MCQ and vision. Usemaxwithmax_tokensβ₯ 131,072 for competition math and GPQA-level science. - Tools:
--tool-call-parser glm47 --enable-auto-tool-choicereturns structuredtool_calls. - Images and videos use standard OpenAI multi-part content (
image_url/video_url).
Evaluation protocol
Limitations
- This is an unofficial derivative, not produced or endorsed by Z.ai or NVIDIA.
- Chinese knowledge (C-Eval 78.3 %) and vision (MMMU 70.7 %) are below E224 and E256. Prefer E256 for Chinese and vision workloads.
- It doesn't fit a single DGX Spark. The 128 GB unified memory is smaller than the 124 GiB of weights plus runtime overhead. You need two Sparks or a β₯180 GB GPU.
- MTP serving needs the vLLM patch in
scripts/. - Vision was validated on MMMU only.
- Like the base model, it can produce inaccurate, biased or unsafe content. Evaluate it for your use case before deploying.
License: MIT (inherited from the base model).