πͺ Qwopus3.6-27B-v2
SFT Release
Reasoning-Enhanced Dense Language Model Fine-Tuned on Qwen3.6-27B
𧬠Trace Inversion & Negentropy
π§ 27B Parameters
π₯ 3-Stage Curriculum SFT
π οΈ Vision & Tool-use Support
π‘ What is Qwopus3.6-27B-v2?
πͺ Qwopus3.6-27B-v2 is a reasoning-enhanced dense language model built on top of Qwen3.6-27B. By leveraging a multi-stage curriculum learning pipeline and augmented with Trace Inversion datasets (claude-opus-4.6/4.7-traceInversion), it reverse-engineers the compressed "Reasoning Bubbles" of commercial LLMs into structured, step-by-step synthetic reasoning traces, successfully eliminating logical shortcuts and knowledge fractures.
𧩠Structured Reasoning
Injects reconstructed deep CoT chains to eliminate logical shortcuts via Trace Inversion.
πͺΆ Style Consistency
Enforces strict constraints on the format and convergence of <think> tags.
π Distillation Alignment
Ensures high-quality cross-source SFT data alignment to narrow the capacity gap.
β‘ RL Scalability
Sets up a stable formatting pipeline optimized for downstream Reinforcement Learning (RL).
π‘ 1. Base Model, Training Library & Cooperation
π§ 1.1 Base Model Specifications (Qwen3.6-27B)
Qwen3.6-27B is a state-of-the-art dense large language model developed by Alibaba Cloud. Boasting 27 billion parameters, this base model natively supports long-context modeling and is engineered for agentic workflows, complex logical reasoning, and multimodal capabilities.
π§ͺ 1.2 Hardware Cooperation & Joint Collaboration
This project is built in close collaboration and joint effort with engineer
Kyle Hessling to achieve stable fine-tuning of our 27B parameter model.
π
You can follow him for hardware and model training updates on X / Twitter: @KyleHessling1
π¦₯ 1.3 Fine-tuning Framework (Unsloth)
Our model training is accelerated by the cool library from
Unsloth. Special thanks to the Unsloth team for their highly efficient, memory-optimized training framework.
π
You can visit their documentation for more fine-tuning guidance: unsloth.ai/docs
βοΈ 1.4 Custom MTP Heads Processing & Automation Tooling
This release features a custom splitting and merging methodology designed specifically for Qwen series Multi-Token Prediction (MTP) heads. The automation skill and complete processing pipeline scripts are open-sourced in
qwen-mtp-gguf.
πIf you find this toolkit helpful, please support the project by leaving a star on GitHub!
[!TIP]
Vision & Tool Calling Support: Qwopus3.6-27B-v2 natively supports vision and tool-use capabilities. To enable vision functionality, download mmproj.gguf from the GGUF Repository and place it in the same directory as the main .gguf file.
[!WARNING]
Community Release Notice: Qwopus3.6-27B-v2 is an experimental community release and has not undergone complete safety evaluations or standard benchmarking. It is intended solely for research and exploration.
π 2. Background & Motivation
β οΈ 2.1 The Distillation Dilemma of "Reasoning Bubbles"
In the field of reasoning distillation, commercial closed-source models (such as GPT-4o or Claude 3.5 Sonnet) typically hide their raw step-by-step thinking processes, displaying only highly compressed "Reasoning Bubbles" to users. Direct imitation of these summaries introduces a severe "Information Entropy Trap"βstudent models struggle to mimic jumpy, step-skipped conclusions without undergoing the underlying logic derivations, resulting in reasoning fractures and poor generalizability.
𧬠2.2 Trace Inversion & Negentropy Reconstruction
To address this challenge, we introduce
Trace Inversion datasets to reconstruct the full reasoning pathway. By using a specialized logical reconstructor,
Trace-Inverter-4B, we reverse-engineer the compressed reasoning bubbles into a complete, step-by-step
Learnable Chain-of-Thought (CoT).
This model integrates:
- claude-opus-4.6-traceInversion-9000x: 9,000 high-value, fully reconstructed step-by-step reasoning trajectories.
- claude-opus-4.7-traceInversion-5000x: 5,000 complex multi-turn logic and mathematics samples optimized for negative entropy reconstruction.
This ensures the student model learns continuous, rigorous logical derivations rather than shortcut-ridden summaries.
β‘ 3. Reasoning Efficiency & MTP Speedup
β‘ Reasoning Efficiency & MTP Acceleration
A compact view of how many output tokens are needed to produce correct answers, and how the MTP variant improves inference throughput.
π MTP Speedup
1.66x
Qwopus3.6-27B-v2-MTP reaches 1.66x the official Qwen3.6 speed
π§ Correct-Answer Tokens
918.7
Average tokens when Qwopus answers correctly
π Total Token Reduction
15.0%
627,325 vs 738,238 output tokens
β
Token Conversion
4.64
Correct answers per 10,000 output tokens
Efficiency framing: These measurements are based on parsed outputs from the same 350-question MMLU-style evaluation set. The goal is not only to compare accuracy, but also to measure how much reasoning text each model spends to reach correct answers.
π― 3.1 Correct-Answer Token Cost
π 3.2 Token Conversion Efficiency
𧩠3.3 Chain-of-Thought Length Comparison
π 4. Evaluation & Benchmarks
π Evaluation & Performance Metrics
Detailed benchmark results on MMLU-Pro, SWE-bench, frontend page layout generation, creative coding, and agentic reasoning.
π MMLU-Pro Subset
87.43%
+2.57 pp vs Qwen3.6-27B
π SWE-bench Verified
75.25%
152 / 202 Resolved (Win)
π¨ Web Design
100%
5 / 5 Pages Validated
β‘ Speed (RTX 5090)
43.9
Avg tok/s (Q5_K_M)
π 4.1 MMLU-Pro Selected Subset
Evaluation format: All models in this MMLU-Pro comparison were tested as unquantized, full-format 16-bit checkpoints. The selected evaluation subset covers 7 categories, with 50 questions per category and 350 questions in total.
Summary: On the selected 350-question MMLU-Pro evaluation set, Qwopus3.6-27B-v2 achieved 87.43% accuracy, outperforming Qwen3.6-27B at 84.86%. Qwopus3.6-27B-v2 is stronger in Business, Computer Science, Physics, and Chemistry, while Qwen3.6-27B remains ahead in Mathematics and Health.
Scope note: due to limited resources, only randomly sampled questions from MMLU-Pro were evaluated. To keep the evaluation fair and transparent, model response logs will be placed in the repository test_data folder.
π» 4.2 SWE-bench Verified (controlled-202 slice)
Execution Details: 19h 29m wall-clock on a single RTX 5090 using a 160K fp16 context window. Every instance successfully exited with Submitted status. 0 step-limit hits and 0 context-overflow failures occurred. Median trajectory length was 67 / 275 steps.
β‘
Run agentic harnesses hot: Raising sampling temperature to temp=1.0 with thinking-on effectively eliminates the reasoning-loop failure mode where earlier finetunes hit 78 empty patches. Greedy decoding (temp=0.1) forces the finetune to over-deliberate and loop inside the <think> block, whereas a higher temperature enables the model to utilize the full breadth of reasoning paths established during training.
β‘ 4.3 Throughput & VRAM Allocation (RTX 5090)
Architecture Trade-off: The MoE wins on raw throughput by ~3.7x due to its A3B routing pattern. However, the Dense 27B model offsets this with superior per-token reasoning depth. We recommend the Dense 27B model for complex agentic workflows, long-context reasoning, and code execution, and the MoE model for fast, high-throughput generations. Tight throughput variance (Β±0.75 tok/s) indicates the dense model is fully memory-bandwidth-bound.
π¨ 4.4 Web Design, WebGL Canvas, and Agentic Task Breakdown
π¨ Web Design Layout Generation (All 5 validated end-to-end)
π Canvas / WebGL Creative Coding (Selected Publications)
*Note: Mandelbulb and Three.js crystal scene outputs are excluded due to aesthetic standards and parked in excluded-canvas/.
π§ Agentic Tasks (5 prompts + 1 structured extraction nothink rerun)
- code_debug: Successfully caught all 4 bugs (sort order,
= vs ==, useless loop, off-by-one errors).
- self_critique: Followed the structured instruction loop (INITIAL β CRITIQUE β IMPROVED) and optimized a palindrome algorithm to O(nΒ²) expand-around-center.
- multi_step_planning: Designed a robust 10-step deployment plan with Dockerfile hand-off and explicit pip dependencies.
- tool_use_json: Resolved a 3-tool sequence (
search_flights, book_hotel, get_weather) with completely valid argument shapes.
πΊοΈ 5. Training & Data Pipeline Overview
The training process fuses Trace Inversion data augmentation with a Three-Stage Curriculum Learning pipeline. The core engineering focuses on expanding context length gradually while training on reconstructed reasoning traces to guarantee format stability.
π― 6. Three-Stage Curriculum Learning
To steadily scale up the reasoning quality under long-context inference, Qwopus3.6-27B-v2 adopts a Curriculum Learning strategy, progressively mixing longer and more complex reasoning templates:
π¨ 7. Trace Inversion Case Studies (5 Key Domains Showcase)
To demonstrate how Trace Inversion reconstructs logical continuity and eliminates negative entropy, the following interactive panels show the contrast between raw compressed "Reasoning Bubbles" and the fully step-by-step reconstructed chain-of-thought (Learnable CoT) under 5 typical scenarios:
π Domain 1: Mathematics (Probability Calculation)
π
Case 1: Probability Calculation
Math Case
π¬ Query
A bag contains 4 red balls and 6 blue balls. If two balls are drawn at random without replacement, what is the probability that both are red?
π Compressed Reasoning Bubble
First draw red: 4/10. Second draw red: 3/9. Multiply: (4/10) * (3/9) = 12/90 = 2/15.
β
Final Answer
The probability that both balls drawn are red is 2/15.
2/15
π‘ Synthesized Reasoning Trace (Learnable CoT)
STEP 1
Define
Identify total items and counts: Total balls = 4 (red) + 6 (blue) = 10 balls. The goal is to compute the probability of drawing two red balls in succession without replacement: P(R1 β© R2).
STEP 2
First Draw
Calculate probability of the first event (drawing a red ball first): P(R1) = Number of red balls / Total balls = 4/10. This simplifies to 2/5.
STEP 3
Second Draw
Calculate probability of the second event given that the first was red: Since there is no replacement, there are now 9 balls left in the bag, and 3 of them are red. P(R2 | R1) = 3/9 = 1/3.
STEP 4
Multiply
Apply the multiplication rule for dependent events: P(R1 β© R2) = P(R1) Γ P(R2 | R1). Substitute values: P(R1 β© R2) = (4/10) Γ (3/9) = 12/90.
STEP 5
Simplify
Simplify the resulting fraction: 12/90 can be divided by 6 in both numerator and denominator: 12Γ·6 = 2, 90Γ·6 = 15. The final probability is 2/15.
π
Conclusion: The probability of drawing both red balls is 2/15 (~13.33%).
π Domain 2: Physics (Kinematics)
π
Case 2: Kinematics Formula
Physics Case
π¬ Query
A car starts from rest and accelerates uniformly at 3 m/sΒ² for 6 seconds. How far does the car travel during this time?
π Compressed Reasoning Bubble
Use kinematics equation d = v0*t + 0.5*a*t^2. Initial velocity v0 = 0, a = 3, t = 6. Calculate 0.5 * 3 * 36. Output 54 meters.
β
Final Answer
The car travels 54 meters.
Answer: 54 meters
π‘ Synthesized Reasoning Trace (Learnable CoT)
STEP 1
Goal
Identify the given variables and goal: Calculate the displacement (d) traveled under uniform acceleration from rest.
STEP 2
Extract
Identify given values: Initial velocity (v0) = 0 m/s (starts from rest), acceleration (a) = 3 m/sΒ², and duration (t) = 6 seconds.
STEP 3
Match Formula
Select the kinematic formula relating displacement to acceleration and time: d = v0t + Β½at2.
STEP 4
Compute
Substitute the values: d = (0)Γ(6) + Β½Γ3Γ(6)2. Simplify: first term is 0; 62 = 36; 3 Γ 36 = 108; Β½ Γ 108 = 54 meters.
STEP 5
Verify
Verify using average velocity method: final velocity vfinal = v0 + at = 0 + 3Γ6 = 18 m/s. Average velocity vavg = (0 + 18)/2 = 9 m/s. Distance d = vavg Γ t = 9 Γ 6 = 54 meters. The calculation is robust.
π
Conclusion: The car travels 54 meters during this uniform acceleration period.
π» Domain 3: Coding (Algorithm Logic)
π»
Case 3: Algorithm Logic
Coding Case
π¬ Query
Given an array of integers, find the sum of all even numbers.
π Compressed Reasoning Bubble
Filter evens using x % 2 == 0, then sum them up. Iterate through array, checking modulus.
β
Final Answer
Here is the Python implementation:def sum_even_numbers(arr):
return sum(x for x in arr if x % 2 == 0)
π‘ Synthesized Reasoning Trace (Learnable CoT)
STEP 1
Iterate
Initialize a tracking sum variable to 0. Prepare to iterate through the array from index 0 to N-1.
STEP 2
Check Even
For each element x, apply the modulus operator: check if x % 2 == 0. If True, the number is even.
STEP 3
Accumulate
If the condition is met, add x to the tracking sum. Otherwise, proceed to the next iteration.
STEP 4
Edge Cases
Check for empty arrays or arrays with no even numbers. In both cases, the returned sum is 0. Negative numbers are handled correctly (e.g. -2 % 2 == 0 is True).
STEP 5
Complexity
Verify complexity: The algorithm runs in O(N) time as it inspects each element exactly once, and O(1) auxiliary space.
π
Conclusion: Summing even numbers completed with O(N) time complexity.
π§ Domain 4: Logical Reasoning (Syllogism)
π§
Case 4: Syllogism & Set Inclusion
Logic Case
π¬ Query
If all philosophers are thinkers, and some thinkers are writers, is it necessarily true that some philosophers are writers?
π Compressed Reasoning Bubble
Class inclusion. 'Philosophers' is a subset of 'Thinkers'. 'Writers' overlaps with 'Thinkers', but the overlap could lie entirely outside the 'Philosophers' subset. Thus, it does not necessarily follow.
β
Final Answer
No, it does not necessarily follow.
Answer: No, it does not follow
π‘ Synthesized Reasoning Trace (Learnable CoT)
STEP 1
Set Definition
Define the sets: P = Philosophers, T = Thinkers, W = Writers. Transcribe premises: Premise 1 is P β T. Premise 2 is T β© W β β
.
STEP 2
Objective
Determine if the conclusion P β© W β β
is a logical necessity under all models.
STEP 3
Venn Analysis
Visualize via Venn diagrams. Circle P is completely inside Circle T. Circle W intersects Circle T. W can intersect only the region of T that excludes P. In this layout, P β© W = β
.
STEP 4
Counterexample
Create a semantic counterexample: Let T = Humans, P = French people, W = Japanese people. All French people are Humans. Some Humans are Japanese. Does it follow that some French people are Japanese? No.
STEP 5
Conclusion
Since a valid counterexample exists, the argument forms an invalid syllogism (specifically, Fallacy of the Undistributed Middle). The conclusion is not guaranteed.
π
Conclusion: The syllogism is invalid; the conclusion does not necessarily follow.
π‘ Domain 5: Core Theory (Reasoning Bubble vs. Learnable CoT)
π‘
Case 5: Proprietary API "Reasoning Bubbles" vs. Reconstructed "Learnable CoT"
Core Theory
π¬ Core Problem
What is the difference between proprietary API "Reasoning Bubbles" and reconstructed "Learnable Chain-of-Thought (CoT)"? Why is direct fine-tuning on Reasoning Bubbles problematic?
π Compressed Reasoning Bubble
Reasoning bubbles are condensed summaries of thinking. They hide step-by-step logic, making direct imitation hard for small models. Reconstructed CoT fills in missing steps, providing a smooth learning gradient for model distillation.
β
Final Answer
Proprietary "Reasoning Bubbles" serve as compressed, high-level summaries that cause cognitive gaps in student models. "Learnable CoT" is a step-by-step, fully-derived logical chain that restores intermediate gradients, enabling effective reasoning distillation.
Bubble = What was thought; CoT = How it was thought
π‘ Synthesized Reasoning Trace (Learnable CoT)
STEP 1
Define
Define the Concepts: Reasoning Bubbles are high-level summaries written for humans, whereas Learnable CoT consists of explicit, step-by-step derivations optimized for model learning.
STEP 2
Entropy
Information Loss (Entropy Trap): API summaries hide intermediate calculation steps. Direct imitation introduces severe "information entropy traps"βstudent models are forced to skip logical steps, causing logical confusion.
STEP 3
Gradient
Causal Gradients: Reconstructed CoT unfolds every derivation, calculation, and code design block. By providing continuous token-level supervision, it creates stable logical gradients for model weights to converge on.
STEP 4
Inversion
Trace Inversion Process: Using the problem and the final answer as hard endpoints, and the reasoning bubble as clues, the inversion model reverse-engineers a complete, rigorous reasoning path (Learnable CoT).
π
Conclusion: Reconstructed CoT turns dark black-box data into open, highly learnable logic gradients.
π Context Length and Long-Context Usage
During fine-tuning, this model was trained with a maximum sequence length of 32K tokens. The training data mixture was also constructed around samples up to 32K tokens, so the "Context Length Distribution" shown in this model card reflects the fine-tuning data distribution rather than a hard architectural limit.
The model still inherits the native long-context capability of the Qwen3.6 base model. Therefore, longer context windows such as 128K or 256K may be available in compatible inference runtimes, depending on the backend and configuration.
For practical long-context inference beyond 32K, especially when using llama.cpp / GGUF, it is recommended to enable RoPE/YaRN scaling instead of only increasing n_ctx / --ctx-size. Directly setting a larger context window without RoPE scaling may work in some cases, but it can be less stable and may not achieve the expected long-context performance.
This is consistent with Qwen community guidance for long-context GGUF usage: 128K context generally requires YaRN/RoPE scaling, and it is not necessarily enabled by default in llama.cpp. For example, Qwen maintainers have noted that "128K context length needs YaRN" and that it should be explicitly enabled when supported by the runtime.
Reference: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct-GGUF/discussions/2
Community feedback also suggests that RoPE/YaRN scaling can improve long-context stability for this model family. One user reported that, on HermesAgent-20, Qwopus3.6-35B-A3B-v1 performed better when extending from 32K to 128K via RoPE scaling than when directly setting a 128K context window without scaling, with scores of 83 vs. 72 in their setup. This result may vary depending on the backend, quantization type, KV cache settings, hardware, and benchmark configuration, but it is consistent with the recommendation to use RoPE/YaRN scaling for contexts beyond 32K.
Example llama.cpp configuration for extending from 32K to 128K:
For 256K context, users may need to adjust the scaling factor accordingly and validate the result in their own workload:
Please note that long-context behavior may vary depending on the inference backend, quantization type, KV cache settings, available memory, and task type. For best results, users should benchmark their own target workload when using contexts beyond 32K.
π€ 8. Collaboration & Training Details
This model is a collaborative milestone achieved with hardware engineer Kyle Hessling. You can follow him on X / Twitter: @KyleHessling1 to keep up with the latest hardware infrastructure and distributed training updates. π
β οΈ 9. Known Training & Deployment Issues (IMPORTANT)
While the 27B dense model architecture is relatively stable, certain low-level framework compatibility issues may still surface during large-scale parameter updates and complex long-context training. It is highly recommended to monitor the following technical risk points during secondary fine-tuning and deployment:
[!CAUTION]
Local Fine-Tuning & Deployment Warning: If you attempt to run secondary fine-tuning or merge adapter weights locally, please proceed with caution and be prepared to manually patch model definition files or pin dependency versions strictly.
π 10. Resources & Guides
π GitHub Repository: Jackrong-llm-finetuning-guide
Access the repository to dive into the codebase and reproduce our results locally or on Google Colab.
π 11. Acknowledgements
Special thanks to:
- The Qwen team for providing the powerful Qwen3.6 base model.
- Unsloth for providing the highly efficient fine-tuning framework.
- Open-source datasets and community contributors.
- Kyle Hessling for the close collaboration on this project.
π 12. Citation