empero-ai/Qwen3.8-35B-A3B-Distill

text-generationtransformersUpstream checksumstransformerssafetensorsqwen3_5_moeimage-text-to-textempero-aiqwen3.6

License: apache-2.0
Size: not mirrored yet
Downloads in the last 30 days: 0 / 30 days

No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.

Cataloged from empero-ai/Qwen3.8-35B-A3B-Distill

Qwen3.8-35B-A3B

Developed by Empero

[!Note] This repository contains model weights and configuration files in the Hugging Face Transformers format.

These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, and other standard runtimes with Qwen3.6 architecture support.

Qwen3.8-35B-A3B is a distillation of the Qwen3.8 frontier models into the Qwen3.6-35B-A3B Mixture-of-Experts architecture. The student was trained on curated teacher traces from our internal Qwen3.8 distillation datasets — dense chain-of-thought spanning mathematics, code, general reasoning, instruction following, and tool use, quality-filtered before training.

The objective: bring the reasoning behavior of frontier-scale teachers into a sparse 35B that activates only 3B parameters per token and deploys on a single GPU.

Highlights

Distilled chain-of-thought — every answer opens with a <think> block learned directly from Qwen3.8 teacher traces rather than synthetic self-generated reasoning.
Mathematics and code emphasis — the trace mix is deliberately weighted toward hard math and competitive programming, the domains where distillation moves the needle most at this scale.
Sparse MoE efficiency — 35B total parameters, ~3B active per token; 256 experts with 8 routed per token.
Attention and experts both adapted — our internal MoE training pipeline updates the attention path and the routed and shared expert stacks, not just attention.
Native function calling per Qwen3.6's specification — no wrapper or tool-specific fine-tune required.
262,144-token native context, inherited from the Qwen3.6 base.

Model Overview

Type: Causal Language Model (text path of a vision-language base)
Base: Qwen/Qwen3.6-35B-A3B
Number of Parameters: 35B total / ~3B active per token
Architecture: 40 layers, 256 experts, 8 experts per token, hybrid linear + full attention
Training: SFT (off-policy distillation) on curated teacher traces via our internal MoE training pipeline
Teachers: Qwen3.8 2.4T A95B and Qwen3.8 Flash Next (internal distillation datasets)
Context Length: 262,144 natively

Benchmark Results

Measured with lm-evaluation-harness, HF backend, bfloat16, identical settings and seed for base and student. Zero-shot, loglikelihood scoring.

Task	Metric	Qwen3.6-35B-A3B (base)	Qwen3.8-35B-A3B	Δ
MMLU (57 subjects)	acc	0.838	0.834	−0.004
ARC-Challenge	acc	0.548	0.582	+0.034
ARC-Challenge	acc_norm	0.548	0.591	+0.044
ARC-Easy	acc	0.819	0.830	+0.011
ARC-Easy	acc_norm	0.717	0.766	+0.048

The MMLU difference is within noise (standard error 0.003 on each measurement). The ARC gains are outside it.

Quickstart

python

from transformers import AutoModelForCausalLM, AutoTokenizerimport torch
model_id = "empero-ai/Qwen3.8-35B-A3B-Distilled"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "A snail is at the bottom of a 10-meter well. Each day it climbs 3 meters, each night it slips back 2. How many days until it escapes?"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True,                                 return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=16384,                     temperature=0.6, top_p=0.95, top_k=20, do_sample=True)print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

A recent transformers release with Qwen3.6 support is required, along with the Gated DeltaNet kernels (flash-linear-attention and a CUDA-matched causal_conv1d build) — without them the linear-attention layers fall back to slow, memory-hungry PyTorch ops.

AutoModelForCausalLM loads the text path (34.7B parameters). The vision tower is retained in the checkpoint and is reachable via AutoModelForImageTextToText.

Best Practices

Sampling: temperature=0.6, top_p=0.95, top_k=20. Greedy decoding on long generations is a known repetition-loop failure mode for reasoning models in this class.
Output length: allow generous max_new_tokens (16,384 recommended); every answer opens with a <think> block. Parse and strip the <think>...</think> span for end users.
Scope: the model learned from teacher traces, not from its own rollouts — it inherits the teachers' reasoning style, including occasional over-long deliberation on easy questions.

Limitations

Shorter responses. The student was trained on 8,192-token examples and produces noticeably shorter outputs than the base. Long chains of thought are more likely to be cut short, so behaviour on long-form generation and long-context workloads may be degraded relative to the base.
A v2 is in training with longer-context support, aimed squarely at the point above.
Vision is untouched. The fine-tune is text-only; vision behaviour is inherited from the base and was not evaluated.

Stay in the loop

Support / Donate

If this model helped you, consider supporting the project:

BTC: bc1qx6zepu6sfkvshgdmc4ewu6pk6rpadvpgffpp7v
LTC: ltc1qv2mefzps2vtjcpwfx8xxdrpplrcvltswm68r7x

Provenance & licensing

Weights are released under Apache-2.0, inherited from the Qwen3.6-35B-A3B base. Shared for research and experimentation, as-is.

Acknowledgements

Developed and released by Empero
Base model: Qwen3.6-35B-A3B (Alibaba Qwen team)
Teachers: Qwen3.8 2.4T A95B and Qwen3.8 Flash Next
Training: TRL + Transformers
Linear-attention kernels: flash-linear-attention, causal_conv1d
Evaluation: lm-evaluation-harness (EleutherAI)
Thanks to everyone who used our free community endpoint — a portion of the prompts used to generate this model's synthetic training data came from those interactions.