Qwen3.5-9B - GGUF High-Quality Quantizations
This repository provides GGUF quantized versions of the Qwen/Qwen3.5-9B model, optimized for local execution using llama.cpp and compatible ecosystems.
📌 Version Notes
All quantizations were generated from the official FP16 weights.
- Target: Efficient execution on consumer hardware, mobile/edge devices, and systems with limited memory.
- Performance: The output quality (reasoning, coherence, and accuracy) is strictly dependent on the base model's parameter scale (9B).
📊 Quantization Table
🛠️ Technical Details
- Quantization Date: 2026-03-07
- Tool used:
llama-quantize (llama.cpp)
- Method: K-Quantization (optimized for AVX2/AVX-512 and modern GPU architectures).
🚀 How to Use
Start a local OpenAI-compatible server with a web UI:
llama.cpp (CLI) using model from HuggingFace
llama.cpp (CLI) using downloaded model
llama.cpp (SERVER) using model from HuggingFace
llama.cpp (SERVER) using downloaded model