orcarouter/OrcaSAQ-2-27B
text-generationvllmVerified checksumsqwenqwen3.8qwen3_5orcasaq2quantizationmixed-precision
- License
- apache-2.0
- Size
- 11 GB
- Revision
- main
Mirrored from orcarouter/OrcaSAQ-2-27B
All models by orcarouter β
OrcaSAQ2 27B
High-fidelity 3-bit Qwen3.8 for long-horizon agents.
54 GB β 12.3 GB Β· +0.02% PPL Β· 93.2% Top-1 Agreement Β· 0.031 KLD Β· 262K Context
OrcaRouter AI Gateway Β· X Β· Discord Β· GitHub Β· All Models
27B reasoning. 12.3 GB.
OrcaSAQ2 27B compresses Qwen3.8-27B from a 54 GB BF16 checkpoint to 12.3 GB while preserving extremely high fidelity to the original model.
Built for: long-horizon agents Β· coding Β· tool use Β· reasoning Β· stateful execution
OrcaSAQ2 is a proprietary sensitivity-aware mixed-precision quantization system developed by OrcaRouter and its research team behind.
It is optimized around one goal: Preserve as much useful model behavior as possible inside a practical GPU memory envelope.
The resulting checkpoint provides:
- 77.2% smaller storage footprint
- only +0.02% perplexity versus BF16
- 93.2% token-level Top-1 agreement
- 0.031 mean KLD
- 262K context
- thinking mode
- tool calling
- MTP speculative decoding
- production serving through vLLM
At a glance
4.4Γ smaller. +0.02% perplexity.
The point is not 3-bit.
The point is what survives at 3-bit.
BF16 fidelity
All numbers below are measured using these exact OrcaSAQ2 weights against the BF16 reference through the same evaluation path.
WikiText-2
16,376 predicted tokens
Perplexity
Top-1 agreement
Model footprint
77.2% smaller.
Long-horizon agents
Low-bit fidelity matters more as the horizon grows.
Short benchmarks can hide small degradation.
Agents cannot.
A small model error can change a tool call.
That changes the environment state.
The changed state affects every decision that follows.
Across long trajectories, small errors can compound into large behavioral differences.
That makes long-horizon execution an especially useful stress test for compressed reasoning models.
OrcaSAQ2 performs strongly on long-horizon workloads relative to models in its deployment and parameter class, despite operating from a 12.3 GB checkpoint.
This makes it particularly suitable for:
- coding agents
- terminal agents
- browser agents
- computer-use agents
- security agents
- repository-scale tasks
- multi-tool workflows
- failure recovery
- long-running stateful execution
Why this matters
Perplexity asks:
How similar is the next-token distribution?
Long-horizon evaluation asks:
Can the model still finish the job after many decisions?
For agent models, both matter.
Long-horizon performance
Agent benchmarks depend heavily on the surrounding scaffold, tools, reasoning budget, timeouts and execution environment. The results below are therefore shown as public reference points, not direct apples-to-apples comparisons.
SWE-bench Verified
70.0% SWE-bench Verified from a 12.06 GB 27B checkpoint.
Terminal-Bench 2.1
58.4% Terminal-Bench 2.1 while fitting in ~12 GB of checkpoint storage.
Public scores use different agent stacks and should not be interpreted as a strict model-only ranking.
Architecture
Production serving
Up to 90.1 tok/s single-stream on a 16 GB GPU
Measured under a 15.7 GiB GPU memory cap.
MTP speculative decoding
+38% single-stream decode throughput
MTP trades additional compute and KV capacity for stronger interactive decode performance.
It is particularly useful for:
- coding assistants
- interactive agents
- terminal agents
- tool-heavy applications
- low-concurrency inference
For highly batched workloads, benchmark both configurations.
27B on a 16 GB GPU
OrcaSAQ2's checkpoint is 12.3 GB.
That makes deployment possible on hardware that cannot hold the original 54 GB BF16 checkpoint.
Actual usable memory depends on:
- vLLM overhead
- KV-cache configuration
- MTP
- batch size
- context length
- CUDA graph configuration
A practical starting point for a 16 GB GPU is approximately 32K interactive context, then tune based on the workload.
The model architecture supports up to 262K context.
Built for agents
Long-horizon execution
Coding
Repository-scale generation, editing, testing and debugging.
Tool use
Structured workflows where action-selection quality matters.
Reasoning
Preserving the capabilities of the 27B base model under an aggressive deployment constraint.
Single-GPU deployment
A 12.3 GB checkpoint designed around practical inference hardware.
Production serving
vLLM + MTP + OpenAI-compatible APIs.
Generation samples
One prompt each, first attempt.
Pelican on a bicycle
The standard SVG test, asked for as an animation.
Chain over the chainring, cranks 180Β° out of phase, parallax background. Pure SMIL, no JavaScript. Used as generated.
Low-poly Statue of Liberty
Create a html low-poly 3D models of the Statue of Liberty
A single self-contained HTML file: Three.js scene, orbit controls, procedural geometry.
Quickstart
Install
Download
Serve with vLLM
OpenAI-compatible API
Recommended sampling
Thinking mode is enabled by default.
For agent deployments, benchmark against the actual tool schema, context distribution and reasoning budget used in production.
Evaluation philosophy
A low-bit reasoning model should not be judged by checkpoint size alone.
We look at the intersection of:
Footprint Γ BF16 Fidelity Γ Capability Γ Long-Horizon Stability Γ Serving Performance
A useful low-bit model must remain useful after compression.
Why perplexity alone is not enough
Perplexity is useful and reproducible.
It is not a complete measure of agentic capability.
Quantization can affect:
That is why OrcaSAQ2 reports BF16 fidelity metrics alongside downstream and long-horizon evaluation.
Method
OrcaSAQ2 uses a proprietary sensitivity-aware mixed-precision quantization system developed by OrcaRouter.
The implementation is optimized to preserve model quality under a strict deployment-memory target.
Detailed quantization methodology, calibration strategy, precision allocation and packing techniques are not currently disclosed.
Limitations
- OrcaSAQ2 inherits the capabilities, biases and limitations of Qwen3.8-27B.
- Quantization is not mathematically lossless.
- 93.2% Top-1 agreement means some token decisions differ from BF16.
- +0.02% PPL is a model-fidelity measurement and does not guarantee identical downstream performance.
- Long-horizon comparisons should use a controlled same-harness evaluation.
- This checkpoint is text-only.
- The vision tower is not included.
- OrcaSAQ2 requires the OrcaSAQ2 vLLM integration.
- Maximum architectural context does not imply that the full context fits into every GPU memory envelope.
Open source from OrcaRouter
OrcaCode Review
Open multi-model code review.
OrcaReplay
Record, replay, fork and debug AI-agent runs.
OrcaRouter Lite
Self-hosted multi-model AI infrastructure.
Open model. Open harness. Open bill.
Citation
License
Apache-2.0
Inherited from:
Quantization does not change the underlying license obligations.
One Gateway. Every Model.
Route Smarter Β· Ship Safer Β· Spend Less