Edge0/Edge0-35B-A3B-preview
text-generationmlxVerified checksumsmoeedge-inferenceprerouterlorassd-offloadmlx
- License
- apache-2.0
- Size
- 18 GB
- Revision
- main
Mirrored from Edge0/Edge0-35B-A3B-preview
All models by Edge0 →
[Image blocked: edge0]
Edge0-35b-a3b Preview
A 35B-class sparse MoE that runs in phone-class memory.
3 GiB active memory · 15 tok/s · 4-bit
Edge0-35b-a3b — a 35B MoE LLM that runs at viable speed in under 3 GiB of active memory, via the edge0 streaming inference framework.
Preview status: this is an early preview release of the edge0 pipeline. The checkpoint ships as int4 quantization plus LoRA and prerouter adapters trained for this framework.
Highlights
- Runs in phone-class memory: the full 4-bit checkpoint stays on storage and experts are streamed on demand, so only the active weights are in RAM — under 3 GiB, with no sharding and no upfront download of the weights into memory.
- Fast enough for interactive use: 15 tok/s decode; long prompts fill in at 140 tok/s.
- Quality kept after quantization: Recover-LoRA distillation keeps the int4 model within 3.9 points of its fp16 base.
- Works out of the box: base, LoRA and prerouter adapters ship
together and load automatically via
edge0.
Three mechanisms make this work:
- SSD expert offload: expert weights are streamed from storage on demand — fetched only as routed, so RAM holds just the active weights. Peak memory is bounded by the active set, not the parameter count.
- Prerouter: a trained head predicts expert routing one step ahead, so expert loads overlap the forward pass instead of stalling it — up to +59% decode throughput; the gain grows with storage latency, model size, and routed width K.
- Recover-LoRA: the int4 base is frozen and LoRA adapters are trained by distillation from the FP teacher, recovering most of the quantization loss at 4-bit (see Quality below). Adapters stay unmerged: one read-only base serves multiple adapter sets.
Model summary
The LoRA and prerouter adapters are co-located with the base checkpoint
and load automatically — this repository is a complete, ready-to-run
model directory for edge0.
Quality
All benchmarks were run by us with OpenCompass under identical settings and parameters for both models. The loss of the edge0 pipeline (int4 + adapters) relative to the fp16 base model is small: 3.9 points on average. Max 100:
Performance
Measured with examples/bench.py on a Mac mini M4 Pro, 24 GB:
*Short contexts; long contexts add KV cache. Expert weights stream from SSD on demand and are not resident.
Use cases
- Edge / on-device inference where GPU VRAM is scarce and storage is fast (NVMe, internal flash).
- Batch serving on a single commodity machine — one read-only base serves many LoRA adapter sets without re-quantization.
- Multilingual chat and reasoning with thinking mode enabled by the bundled chat template.
Limitations
- Preview release: coverage and quality are still being extended; the model is primarily tuned for the languages of the base model.
- Agent capability: this preview release is not yet optimized for agentic tasks — tool use, multi-step planning, and long-horizon autonomy are currently weak. The full release will substantially strengthen agent capability.
- The MLX backend currently targets Apple Silicon; other backends are on the edge0 roadmap.
- Long contexts grow the KV cache; use shorter contexts to keep peak memory at 3 GiB.
Quick start
For full usage (Python API, streaming options, prerouter details), see the edge0 documentation.
License
Apache 2.0. See LICENSE.
Citation
If you find Edge0 useful in your research, please cite our paper: