No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ
All models by ISTA-DASLab →Low-bit quantization of Qwen/Qwen3.6-35B-A3B
(MoE, 35B total / 3B active) produced with GSQ
(Gumbel-Softmax Quantization).
The routed-expert MLP weights are quantized to 2-bit GSQ with an effective storage cost of ≈2.13 bpp, while the attention layers, shared experts, and LM head are quantized to INT8. The checkpoint preserves most of the base model's reasoning, coding, and long-context behaviour while substantially reducing its memory footprint.
We evaluate the quantized checkpoint against the original
Qwen/Qwen3.6-35B-A3B
across reasoning, instruction-following, science QA, and math benchmarks.
Qwen/Qwen3.6-35B-A3B{-2, -1, 0, +1} × scaleself_attn, linear_attn, shared experts, and lm_headgate_proj, up_proj, down_proj)self_attn, linear_attn), shared experts, and LM headgate, and other non-quantized componentsThe Hugging Face "Tensor types" widget reports the container dtype of each tensor stored in the safetensors checkpoint, rather than the effective precision of the original model weights.
The routed-expert MLPs use the Humming exact-width 2-bit layout. For every
2-bit expert-MLP Linear with original weight shape
[out_features, in_features], the following tensors are stored:
For the 2-bit GSQ expert weights, the effective storage is:
The checkpoint therefore uses a mixed-precision quantization layout: 2-bit GSQ for routed-expert MLP weights, INT8 for attention, shared experts, and the LM head, and BF16 for the remaining unquantized components.
The quantization_config in config.json describes the Humming quantization
layout used by the checkpoint.
Note: GSQ training first writes shards in
compressed-tensorspack-quantizedformat, where the 2-bit codebook is padded into a 4-bit INT32 container. The published checkpoint has been re-packed viaconvert_to_humming.pyinto exact-width 2-bit Humming storage, hence the2 / 32shape factor for the 2-bit expert weights.
Serving this checkpoint requires:
Install the required vLLM version:
Then, from the directory containing the model files, run the included
patch_vllm.py script:
The patch must be executed in the same Python environment in which vLLM 0.27.1 is installed. It patches the installed vLLM package with the changes required to load and serve this checkpoint.
After applying the patch, the model can be served normally:
Important: running the model with an unpatched vLLM installation is not supported. If vLLM is reinstalled or the environment is recreated, run
python patch_vllm.pyagain before serving the model.
Model size / text-only usage: the full checkpoint is approximately 12.6 GB, including the MTP and vision components. These components are optional for ordinary text-only generation: the vision weights are only required for multimodal inputs, while the MTP weights are only required when using MTP/speculative decoding. Removing both the vision and MTP weights reduces the model size to approximately 9.9 GB.
Actual VRAM usage during serving will be higher than the raw model size and depends on KV-cache allocation, context length, batch size, and vLLM runtime overhead.
2 bits (packed) + 16 bits / 128 (group scale) ≈ 2.13 bpppip install vllm==0.27.1python patch_vllm.pyvllm serve ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ --reasonin-parser qwen3@article{gsq2026, title = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling}, author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurti{\'c}, Eldar and Kleinegger, Max and Alistarh, Dan}, journal= {arXiv preprint arXiv:2604.18556}, year = {2026}, url = {https://arxiv.org/abs/2604.18556}}