Cataloged from Jackrong/Qwopus3.5-9B-Coder-MTP-GGUF
[!NOTE] What is MTP (Multi-Token Prediction)?
MTP is a revolutionary technology in the field of Large Language Model (LLM) training and inference in recent years. Unlike traditional autoregressive models that predict only a single token at each step (Single-Token Prediction), MTP models are designed during training to simultaneously predict multiple future tokens at each position.
This architecture brings two core dimensions of transformation:
- Deeper Representation and Planning: It forces the model to perform global planning (Long-Horizon Planning) at the representation level for longer-term contexts. This enhances cognitive coherence in logic-intensive tasks such as complex coding and multi-step mathematical reasoning, while effectively mitigating the "reasoning bubbles" and repetition loops common in traditional autoregressive models.
- Extreme Inference Speedup (Speculative Decoding): During inference, the model is equipped with additional lightweight auxiliary prediction heads (Draft Heads, configured as
draft=2in this model). While the backbone network generates the current token, the Draft Heads predict the subsequent 2 candidate tokens in parallel with negligible computational overhead, which are then verified by the main model in a single forward pass. Once verified, the model can output multiple tokens in a single inference step, yielding substantial throughput gains.
Based on actual testing across Logic / Coding / DevOps / Math / Edge (5 core domains, 30 complex evaluation questions), Qwopus3.5-9B-Coder-MTP (draft=2) demonstrates absolute advantages in both speed and correctness:
[!IMPORTANT] The evaluation configuration and benchmark framework follow the official Qwen series testing by the Unsloth team, whose research demonstrates that setting
draft=2yields the optimal performance. For full details, see the official Unsloth MTP Benchmarks.
[!TIP] A community user independently evaluated Qwopus3.5-9B-coder on a tool-recall test with up to 31 available tools and an adversarial tool-selection phase containing semantic overlaps and decoy tools.
| Model | Phase 1: Tool Recall | Phase 2: Adversarial Tool Selection |
|---|---|---|
| Qwopus3.5-9B-coder | 100% | 27 / 28 (96%) |
| Claude Opus 4.6 | 100% | 27 / 28 (96%) |
| Qwen3.5-9B | 100% | 26 / 28 (93%) |
To guarantee the rigor, objectivity, and reproducibility of the evaluation, this benchmark was conducted under a unified hardware platform and sampling hyperparameters:
1.0 (recommended standard, balancing logical reasoning and creativity).0.95 (retains high-probability candidates, filters tail noise, ensuring reasoning accuracy).[!TIP] Logic: Base 5/5 = 100% | MTP 5/5 = 100%
[!TIP] Coding: Base 7/10 = 70% | MTP 10/10 = 100%
[!TIP] DevOps: Base 2.5/5 = 50% | MTP 3.5/5 = 70%
[!TIP] Math: Base 4.5/5 = 90% | MTP 5/5 = 100%
[!TIP] Edge: Base 5/5 = 100% | MTP 3/5 = 60%
As the base model of this model, Qwopus3.5-9B-v3.5 is already a model with powerful capabilities. On this foundation, Qwopus3.5-9B-coder is specially optimized and fine-tuned for high-performance 🤖 Agentic Coding, complex Tool Calling, and logical reasoning.
💡 Why the 9B Dense Model? We believe that the 9B dense architecture represents the perfect "sweet spot" for large language models. It runs seamlessly at 8-bit precision on entry-level 16GB RAM devices—such as standard laptops and the Mac mini—making it exceptionally lightweight yet highly versatile. Without requiring expensive hardware, it allows you to achieve excellent performance paired with impressive inference speeds. Simply put, Qwen3.5-9B is currently the best open-source model in its class.
[!TIP] Vision & Tool Calling Support: This model supports visual capabilities and tool calling. To enable vision, please place the
mmproj.gguffile from the GGUF repository into the same directory as the main.gguffile.
The fine-tuning process of this model deeply integrates Trace Inversion data augmentation technology with high-quality Agent Traces. This systematic approach not only strengthens the model's ability to solve complex programming tasks, but also greatly improves its logical coherence and accuracy when using various tools.
This model is designed specifically for the following goals:
[!WARNING]
- Community Release Notice: Qwopus3.5-9B-coder is released purely as an experimental community version, aiming to explore the combination of Agent capabilities and deep reasoning, and is only for research and exploration use.
- Warning: Because this model is vertically fine-tuned for programming agents and deep reasoning, and has not undergone comprehensive general performance evaluation, its capabilities in general domains or specific non-programming tasks may suffer from Capability Decay. Users are advised to be aware of its limitations in other scenarios while exploring its core capabilities.
To verify the execution efficiency and logical robustness of Qwopus3.5-9B-coder in actual agent scenarios, we adopted the open-source testing framework benchlocal.
The following shows the comparative performance on SWE-bench Verified, which evaluates language models on resolving software engineering issues in real-world open-source repositories:
[!IMPORTANT]
- ⚙️ All tests were conducted with a temperature of 1 as officially recommended by qwen3.5. All errors and model issues were attempted to be regenerated twice after a test failure. If both attempts fail, it is considered a failure.
- 🍎 All screenshots of the test interfaces have been uploaded to the image folder in the repository. Click the link below to view and verify:
- 🔗 View Test Screenshots
- ❤️ Kyle Hessling for his generous hardware and equipment support. You can follow him for more updates on X / Twitter: @KyleHessling1.
In order to break through the "reasoning bubble" limitation of the model in actual programming and tool usage, and to endow it with real Agent behavioral capabilities, this model introduced core augmented datasets during training:
Currently, based on public information, commercial models such as OpenAI's GPT series and Anthropic's Claude series have very clearly hidden the true internal reasoning chains of their models. For these models, what we can ultimately see in the API or front-end interface can often only be considered a highly compressed "Reasoning Bubble".
To break through this limitation, we adopted the Trace Inversion technology. This technology utilizes an external "surrogate model" to reconstruct a complete and logically coherent deep reasoning chain based on the "question + final answer + compressed reasoning summary" published by commercial models. The "reasoning bubble", which originally consisted of only a few sentences and logical leaps, is expanded into a high-quality deep learning trace with complete derivation, calculation, and logical verification, providing step-by-step logical learning signals for the model.
To significantly enhance the model's execution and coding capabilities in real environments, this model additionally introduced the lambda/hermes-agent-reasoning-traces dataset.
<think> tags, but also includes actual tool execution results (rather than fabricated outputs out of thin air).By learning these Agent trajectories that contain real feedback and thoughtful processes, Qwopus3.5-9B-coder can exhibit thinking and operational modes closer to human experts when facing complex programming and system operations tasks.
The training of this model integrates a phased learning pipeline of Trace Inversion data augmentation technology and high-quality Agent Trajectories data. Its core logic lies in restoring the highly compressed "reasoning bubble" of commercial models into a deep path for learning, and combining it with real agent operational traces to comprehensively improve the model's logical reasoning and code execution capabilities.
[ 🗺️ Trace Inversion: Full Process of Data Inversion and "Attack" Distillation ]
A. Surrogate Model Training
Open Source Model (GLM-5.1 / DS-V4) ──► Complete Reasoning Chain ──► [ Qwen3-235B Compression ] ──► Reasoning Bubbles
│ │
└──────────► [ Training ] ◄─────────┘
(Base: Qwen3-4B-Instruct)
(Result: Trace-Inverter-4B)
B. Inversion Phase: "Attacking" Claude-4.7-Max
_______________________________________________________
| |
| Claude-4.7-Max API ──► Compressed Bubbles + Final Answer |
|_______________________________________________________|
│
▼
[ 🧠 Trace-Inverter-4B (Logical Reconstructor) ] ────► Synthetic CoT
│
▼
[ 🧩 Data Splicing ] ◄────────── (Original Prompt + Response)
(Embed the inverted chain of thought into <think> tags, and splice with the original Q&A pair for restoration)
│
▼
(Result: claude-opus-4.6/4.7 Inversion Set)
C. Final SFT Pipeline
___________________________________________
| |
| Base Model (Qwopus3.5-9B-v3.5) |
|___________________________________________|
│
▼
[ 📦 Stage 1: Format Establishment and Logic Injection ] ───────► [ 🛠️ Stage 2: Agent Trajectories and Programming Reinforcement ]
(Integrate inverted reasoning data, stabilize thinking format) (Introduce GLM-5.1 Agent Trajectories, reinforce interaction and execution)
│ │
│ ▼
│ __________________________________________________
│ | 🔍 Hermes Agent Trace Sample Structure Breakdown (GLM-5.1) |
│ | 1. [🛠️ System] -> JSON Tool Definition |
│ | 2. [👤 Human] -> Initial Task Instruction |
│ | ┌──────────────────────────────────────────────┐ |
│ | │ 🔁 Multi-turn Loop: │ |
│ | │ 3. [🧠 GPT] -> <think> Logical Reasoning/Reflection │ |
│ | │ 4. [🤖 GPT] -> Tool Call Execution Action │ |
│ | │ 5. [⚙️ Tool] -> Real Feedback │ |
│ | └──────────────────────────────────────────────┘ |
│ |__________________________________________________|
│ │
└────────────────┬────────────────┘
▼
___________________________________
| |
| 🌟 Final Model: Qwopus3.5-9B-coder |
|___________________________________|
[!NOTE] Because agent trajectory datasets are complex and diverse. The datasets have undergone rigorous cleaning and formatting.
Qwopus3.5-9B-coder adopts a phased reasoning data mixture strategy similar to Curriculum Learning, gradually increasing the difficulty and complexity of training signals:
Early Stage (Format Establishment): Focuses on short-to-medium length reasoning samples with stable formats. The primary goal of this stage is to establish a reliable, structured new reasoning format while avoiding overwhelming the model with extreme complexity.
Middle Stage (Complexity Scaling & Multi-Teacher Distillation): Gradually increases the proportion of complex reasoning samples from multiple teacher models.
Late Stage (Long-Context Reinforcement & Drift Prevention): Reinforces reasoning capabilities in long contexts. Crucially, this stage retains short-sample replay to ensure the model maintains its short-context instruction-following capability and minimizes capability drift.
This model is the result of continuous exploration in Agentic AI and reasoning capabilities.
Training Infrastructure & Configuration:
[!CAUTION] Compatibility and Deployment Notice
- Tool Calling Format: When using this model for tool calling, please ensure that you use a Prompt format and System Prompt that match the training data to activate its Agent capabilities.
- Reasoning Output Extraction: The model's thinking process is typically wrapped in
<think>and</think>tags. When deploying to front-end applications, these tags may need to be parsed and hidden.
👉 GitHub Repository: Jackrong-llm-finetuning-guide Visit the repository to dive into our fine-tuning codebase and guides.
Special thanks to:
@misc{jackrong_qwopus35_9b_coder,
title = {Qwopus3.5-9B-coder},
author = {Jackrong},
year = {2026},
publisher = {Hugging Face}
}