autotrust/GLM5.3-Flash-E224-DGX-Spark
image-text-to-textvllmVerified checksumsautotrustglm5_nextmoenvfp4modeloptvllm
- License
- mit
- Size
- 157 GB
- Revision
- main
Mirrored from autotrust/GLM5.3-Flash-E224-DGX-Spark
All models by autotrust βMirrored from autotrust/GLM5.3-Flash-E224-DGX-Spark
All models by autotrust βautotrust/GLM5.3-Flash-E224-DGX-Spark is a compact build of zai-org/GLM-5.3-Flash for desktop Blackwell systems like NVIDIA DGX Spark. It's an unofficial derivative.
π Deploying on DGX Spark? A complete, field-tested runbook for serving this checkpoint on two ConnectX-7-linked DGX Sparks (vLLM 0.31.0, TP=2, fp8 KV cache) is here: DEPLOY-2X-DGX-SPARK.md β network setup, NCCL/CX7 pitfalls, memory budgeting, ops & troubleshooting, and measured throughput (15.7 tok/s single-stream, 71.8 tok/s at 8-way concurrency, both GPUs β96% utilized).
It keeps 224 of the 288 routed experts in each layer by Neural Architecture Search (NAS), uses NVFP4 for the experts and still activates 18 B parameters per token. The weights take 141 GiB, which is small enough for two DGX Sparks connected by ConnectX-7 (256 GB of unified memory in total) with room left for long-context KV cache. A single 180 GB Blackwell GPU (B200/GB200) can also run it.
The original GLM-5.3-Flash MTP layer ships unmodified in mtp/ [blocked] as an optional speculative-decoding draft. It gives about 1.85Γ single-stream decode speed at the same output quality.
The model keeps the full 154,880-token vocabulary and has the vision tower intact.
DGX Spark (GB10 Grace Blackwell) has 128 GB of LPDDR5X unified memory, 273 GB/s bandwidth and native FP4 tensor cores. That profile shaped this build:
reasoning_effort=max.β οΈ Hardware validation status. All accuracy and throughput numbers below were measured on a single NVIDIA B200 with the same weights. The 2Γ DGX Spark deployment recipe below follows NVIDIA's standard two-Spark vLLM setup. Memory figures for Spark are calculated from the measured weight footprint, not measured on Spark hardware. Expect much lower absolute tokens/s on Spark than on B200, because GB10 has about 30Γ less memory bandwidth. Reports from Spark owners are very welcome in the Community tab.
Measured on one B200 with vLLM. Sampling follows the base model's official recipe (temperature=1.0, top_p=0.95); HumanEval also uses greedy decoding. reasoning_effort is the GLM-5.3-Flash chat-template thinking budget (low / high / max).
Scoring is strict: a response that runs out of tokens before giving a final answer counts as wrong. All numbers are single runs. MoE decoding in vLLM isn't bit-deterministic, so treat Β±2β3 points as noise.
At the base model's recommended thinking budget (max), GPQA-Diamond and AIME 2025 match the unpruned NVFP4 checkpoint published by RedHatAI.
Token usage per question (completion tokens, thinking included):
On Spark, size --max-model-len for the effort you use. About 64 K is enough for low. Use β₯ 160 K for max; the long tail of hard problems runs past 130 K tokens.
BFCL was run with bfcl-eval v4 against the local OpenAI-compatible endpoint (--tool-call-parser glm47, template-default effort, --num-threads 32). multi_turn_miss_func, miss_param, long_context and the agentic web-search/memory categories weren't run.
reasoning_effort=low, 1,024 output tokens, short prompts, full CUDA graphs:
These numbers come from a B200 with about 8 TB/s of HBM bandwidth. A DGX Spark has 273 GB/s per node, so single-stream decode there will be much slower. Plan for interactive single-user or small-batch serving on Spark, not high-concurrency throughput.
The mtp/ folder holds the original GLM-5.3-Flash MTP layer, unchanged: BF16, all 288 experts, 17 GB. vLLM loads it as a separate draft model. Speculative decoding is lossless; accuracy measured with MTP on matches the runs without it, within noise.
Measured on a single B200 with reasoning_effort=low, 1,024 output tokens, served from this repository as uploaded. Text, Chinese, tool-calling and image smoke tests pass both with MTP on and off.
ΒΉ Earlier run with identical weight files.
When to use MTP: turn it on for interactive, low-concurrency serving (1β8 streams), which is the typical DGX Spark workload. Turn it off for high-concurrency batch serving. At 32 concurrent streams, MTP lowers aggregate throughput by about 16 % and raises median TTFT from 0.8 s to 6.4 s, because the draft's extra weights shrink the KV cache: on one B200 at --gpu-memory-utilization 0.97, the KV cache drops from 1.31 M to 0.32 M tokens.
With MTP on (2 draft tokens), HumanEval greedy scored 97.0 % and GPQA-Diamond (low) scored 77.8 %, in line with the runs without MTP. Acceptance over the reasoning-heavy eval traffic was 2.36 tokens per step.
MTP is especially useful on DGX Spark. Decode there is memory-bandwidth bound, and single-user interactive use is the typical workload, which is exactly where speculative decoding helps most. The cost is 17 GB of extra weights (about 8.5 GB per node at TP=2), so leave room for it in your memory budget (see Deployment).
max), the gap closes to within noise.vllm/vllm-openai:glm53-flash image. Validated here on the ZJY0516/vllm@glm-release branch (vllm-project/vllm#53906) at commit 7e2d791 plus 8f8cc41 ("Make GLM-5.3 kpool metadata graph-safe"). Without that fix, full CUDA graphs can crash under concurrency.transformers >= 5.16.1VLLM_USE_DEEP_GEMM=0.You can also use the Ray-based run_cluster.sh flow from NVIDIA's dgx-spark-playbooks vLLM guide with --tensor-parallel-size 2.
Memory per node (estimate): about 70.5 GiB of weights + about 40 GiB of KV cache and activations at --gpu-memory-utilization 0.85 (unified memory is shared with the OS). With MTP enabled, add about 8.5 GiB per node for the draft. Lower --max-model-len or raise the utilization if you don't need max effort.
Enable MTP (recommended for interactive use) by adding this to the command on both nodes:
If TP=2 over the interconnect is slow or unstable on your setup, try --tensor-parallel-size 1 --pipeline-parallel-size 2. Pipeline parallelism sends far less traffic over the cable per token, at the cost of single-stream latency.
On a 183 GB B200 this loads 141.5 GiB of weights and leaves about 14β18 GiB of KV cache: 1.1 M tokens with BF16 KV at 172 K context, or 2.5 M tokens with --kv-cache-dtype fp8. With MTP (--speculative-config '{"method": "mtp", "model": "<repo>/mtp", "num_speculative_tokens": 2}'), weights total 155.3 GiB. Use --max-model-len 66560 --gpu-memory-utilization 0.97 on a single B200.
reasoning / reasoning_content field.reasoning_effort: low, high or max (the default). Pass it as the top-level OpenAI field or via chat_template_kwargs.temperature=1.0, top_p=0.95. Greedy decoding can make long thinking loop on hard prompts.low for chat, Q&A, tool calls, MCQ and vision. Use max with max_tokens β₯ 131,072 for competition math and GPQA-level science.--tool-call-parser glm47 --enable-auto-tool-choice returns structured tool_calls.image_url / video_url).License: MIT (inherited from the base model).
# on both nodes: point NCCL/Gloo at the ConnectX-7 interface (check `ibdev2netdev`)export VLLM_USE_DEEP_GEMM=0export NCCL_SOCKET_IFNAME=enp1s0f1np1 GLOO_SOCKET_IFNAME=enp1s0f1np1
# node 0 (head)vllm serve autotrust/GLM5.3-Flash-E224-DGX-Spark \ --served-model-name glm53-flash-e224 \ --tensor-parallel-size 2 --nnodes 2 --node-rank 0 --master-addr <HEAD_CX7_IP> \ --kv-cache-dtype fp8 --max-model-len 163840 --max-num-seqs 8 \ --gpu-memory-utilization 0.85 \ --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \ --host 0.0.0.0 --port 8000
# node 1 (worker)vllm serve autotrust/GLM5.3-Flash-E224-DGX-Spark \ --tensor-parallel-size 2 --nnodes 2 --node-rank 1 --master-addr <HEAD_CX7_IP> --headless \ --kv-cache-dtype fp8 --max-model-len 163840 --max-num-seqs 8 \ --gpu-memory-utilization 0.85 --speculative-config '{"method": "mtp", "model": "<local-path-to-this-repo>/mtp", "num_speculative_tokens": 2}'export VLLM_USE_DEEP_GEMM=0vllm serve autotrust/GLM5.3-Flash-E224-DGX-Spark \ --served-model-name glm53-flash-e224 \ --kv-cache-dtype fp8 --max-model-len 172032 --max-num-seqs 48 \ --gpu-memory-utilization 0.93 \ --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45from openai import OpenAIclient = OpenAI(base_url="http://<head-node>:8000/v1", api_key="EMPTY")r = client.chat.completions.create( model="glm53-flash-e224", messages=[{"role": "user", "content": "Prove that sqrt(2) is irrational."}], temperature=1.0, top_p=0.95, max_tokens=32768, extra_body={"reasoning_effort": "high"},)print(r.choices[0].message.content)