RedHatAI/Qwen3-8B-speculator.dflash2

speculatorsUpstream checksumsspeculatorssafetensorsspeculative-decodingdflash2custom_codearxiv:2405.05612

License: apache-2.0
Size: not mirrored yet
Downloads in the last 30 days: 0 / 30 days

No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.

Cataloged from RedHatAI/Qwen3-8B-speculator.dflash2

RedHatAI/Qwen3-8B-speculator.dflash2

This is a DFlash2 speculator model for Qwen/Qwen3-8B, trained without reasoning (enable_thinking=False).

Training Details

This model was trained using the Speculators library on a mix of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered, the train_sft split of HuggingFaceH4/ultrachat_200k, and nvidia/Nemotron-Personas data. Responses were regenerated by Qwen3-8B (without reasoning). Training uses D-PACE loss weighting and the Muon optimizer, with a re-warm cosine learning rate schedule for continued improvement after initial convergence.

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

Launch vLLM

bash

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \  Qwen/Qwen3-8B \  --target-layer-ids 3 11 19 27 35 43 \  -- --port 8000 \  --gpu-memory-utilization 0.85 \  --tensor-parallel-size 4 \  --disable-uvicorn-access-log

Launch training (resumed with re-warm cosine LR)

bash

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \  --standalone \  --nproc_per_node 4 \  scripts/train.py \  --verifier-name-or-path Qwen/Qwen3-8B \  --speculator-type dflash2 \  --block-size 16 \  --num-layers 8 \  --data-path ./output \  --vllm-endpoint http://localhost:8000/v1 \  --save-path ./output/checkpoints \  --epochs 7 \  --lr 6e-4 \  --total-seq-len 16384 \  --max-anchors 768 \  --target-layer-ids 3 11 19 27 35 43 \  --on-missing generate \  --on-generate delete \  --loss-fn ce \  --per-position-loss-weight dpace \  --dpace-alpha 0.5 \  --optimizer muon \  --muon-lr 0.003 \  --muon-momentum 0.95 \  --muon-weight-decay 0.1 \  --muon-ns-steps 5 \  --weight-decay 0.01 \  --scheduler-type cosine \  --scheduler-total-steps 68922 \  --prefetch-factor 2 \  --num-workers 8 \  --log-freq 100

Model Specifications


Base Model	Qwen/Qwen3-8B
Chat Template	Qwen/Qwen3-8B (use `/chat/completions` endpoint)
Reasoning	Disabled (`enable_thinking=False`)
Format	Safetensors
License	Apache 2.0
Validation Hardware	8× Nvidia H100

Deployment

bash

pip install vllm
vllm serve Qwen/Qwen3-8B \    --tensor-parallel-size 1 \    --max-model-len 16384 \    --speculative-config '{        "model": "RedHatAI/Qwen3-8B-speculator.dflash2",        "num_speculative_tokens": 7,        "method": "dflash2"    }'

Evaluations

Per-position token acceptance rates on the RedHatAI speculative decoding benchmark (native 7 draft tokens, enable_thinking=False, max_tokens=512, temperature=0):

Dataset	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Avg Length
HumanEval	88.0%	75.8%	64.4%	54.5%	46.5%	39.4%	33.4%	5.018
math_reasoning	92.1%	83.3%	75.0%	66.8%	58.8%	52.1%	46.5%	5.747
qa	67.9%	45.3%	29.2%	19.4%	12.4%	8.4%	5.0%	2.876
question	74.3%	53.4%	37.1%	26.9%	20.0%	15.5%	12.0%	3.390
rag	73.6%	52.9%	37.3%	26.1%	18.4%	12.5%	8.5%	3.293
summarization	70.8%	46.3%	28.8%	17.2%	10.4%	6.5%	3.8%	2.838
tool_call	77.0%	57.5%	42.6%	32.7%	23.7%	17.8%	13.6%	3.649
translation	79.6%	57.7%	39.9%	20.9%	10.8%	6.1%	3.8%	3.188
writing	74.9%	53.6%	37.6%	27.3%	20.5%	16.0%	12.4%	3.424

References

Paper: DFlash2: Block Diffusion for Improved Speculative Decoding

Training Details

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

bash

python scripts/prepare_data.py \  --model Qwen/Qwen3-8B \  --data ./regenerated_data.jsonl \  --output ./output \  --seq-length 16384

python scripts/prepare_data.py \  --model Qwen/Qwen3-8B \  --data ./regenerated_data.jsonl \  --output ./output \  --seq-length 16384

Launch vLLM

bash

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \  Qwen/Qwen3-8B \  --target-layer-ids 3 11 19 27 35 43 \  -- --port 8000 \  --gpu-memory-utilization 0.85 \  --tensor-parallel-size 4 \  --disable-uvicorn-access-log

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \  Qwen/Qwen3-8B \  --target-layer-ids 3 11 19 27 35 43 \  -- --port 8000 \  --gpu-memory-utilization 0.85 \  --tensor-parallel-size 4 \  --disable-uvicorn-access-log

Launch training (resumed with re-warm cosine LR)

bash

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \  --standalone \  --nproc_per_node 4 \  scripts/train.py \  --verifier-name-or-path Qwen/Qwen3-8B \  --speculator-type dflash2 \  --block-size 16 \  --num-layers 8 \  --data-path ./output \  --vllm-endpoint http://localhost:8000/v1 \  --save-path ./output/checkpoints \  --epochs 7 \  --lr 6e-4 \  --total-seq-len 16384 \  --max-anchors 768 \  --target-layer-ids 3 11 19 27 35 43 \  --on-missing generate \  --on-generate delete \  --loss-fn ce \  --per-position-loss-weight dpace \  --dpace-alpha 0.5 \  --optimizer muon \  --muon-lr 0.003 \  --muon-momentum 0.95 \  --muon-weight-decay 0.1 \  --muon-ns-steps 5 \  --weight-decay 0.01 \  --scheduler-type cosine \  --scheduler-total-steps 68922 \  --prefetch-factor 2 \  --num-workers 8 \  --log-freq 100

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \  --standalone \  --nproc_per_node 4 \  scripts/train.py \  --verifier-name-or-path Qwen/Qwen3-8B \  --speculator-type dflash2 \  --block-size 16 \  --num-layers 8 \  --data-path ./output \  --vllm-endpoint http://localhost:8000/v1 \  --save-path ./output/checkpoints \  --epochs 7 \  --lr 6e-4 \  --total-seq-len 16384 \  --max-anchors 768 \  --target-layer-ids 3 11 19 27 35 43 \  --on-missing generate \  --on-generate delete \  --loss-fn ce \  --per-position-loss-weight dpace \  --dpace-alpha 0.5 \  --optimizer muon \  --muon-lr 0.003 \  --muon-momentum 0.95 \  --muon-weight-decay 0.1 \  --muon-ns-steps 5 \  --weight-decay 0.01 \  --scheduler-type cosine \  --scheduler-total-steps 68922 \  --prefetch-factor 2 \  --num-workers 8 \  --log-freq 100

Deployment


Base Model	Qwen/Qwen3-8B
Chat Template	Qwen/Qwen3-8B (use `/chat/completions` endpoint)
Reasoning	Disabled (`enable_thinking=False`)
Format	Safetensors
License	Apache 2.0
Validation Hardware	8× Nvidia H100

bash

pip install vllm
vllm serve Qwen/Qwen3-8B \    --tensor-parallel-size 1 \    --max-model-len 16384 \    --speculative-config '{        "model": "RedHatAI/Qwen3-8B-speculator.dflash2",        "num_speculative_tokens": 7,        "method": "dflash2"    }'

pip install vllm
vllm serve Qwen/Qwen3-8B \    --tensor-parallel-size 1 \    --max-model-len 16384 \    --speculative-config '{        "model": "RedHatAI/Qwen3-8B-speculator.dflash2",        "num_speculative_tokens": 7,        "method": "dflash2"    }'

Evaluations

Per-position token acceptance rates on the RedHatAI speculative decoding benchmark (native 7 draft tokens, enable_thinking=False, max_tokens=512, temperature=0):