Mirrored from harshatheg/Qwen-2.5-1B-RLCD
All models by harshatheg →
Live Demo: Try the side-by-side comparison live on Hugging Face Spaces: drinkmoonshine/parallel-constrained-decoding.
A high-throughput inference engine for structured information extraction, decision routing, and categorical classification on Apple Silicon using MLX.
Parallel Constrained Decoding evaluates multi-field JSON schemas simultaneously rather than generating tokens sequentially. On an Apple Silicon M4 Max, it delivers 5.6x to 7.0x latency reductions compared to standard autoregressive decoding with 100% schema validity and calibrated field-level confidence scores.
Evaluated with mlx-community/Qwen2.5-1.5B-Instruct-4bit on macOS Sequoia:
Standard LLM structured generation (such as JSON mode or grammar-guided sampling) relies on token-by-token autoregressive decoding:
Each token requires a distinct GPU/NPU forward pass and sequential memory bandwidth roundtrips. As schema size grows, latency scales linearly with output token length:
Additionally, autoregressive decoding is susceptible to syntax degradation, field omission, and hallucinated keys.
In structured extraction and classification, field values belong to bounded candidate sets (booleans or categorical enums). Parallel Constrained Decoding exploits this property:
Clone the repository and install dependencies:
Schemas are defined using StructuredSchema. Each field specifies a type (enum or boolean), a description to guide model reasoning, and choices (for enum types, supporting up to 255 choices):
You can also construct fields explicitly using FieldDefinition:
Execute parallel constrained inference on your context string:
The output dictionary provides both the structured JSON and detailed field telemetry:
To compare against standard autoregressive generation:
The repository includes a web interface for side-by-side latency and accuracy comparison.
To launch the web server:
Or run directly with uvicorn:
Open http://localhost:8000 in your browser.
Run the benchmark suite across pre-configured enterprise presets:
Output example:
The engine is currently configured for mlx-community/Qwen2.5-1.5B-Instruct-4bit.
Any decoder LLM supported by mlx-lm can be loaded by setting MODEL_ID in core/engine.py.
Apache 2.0
[Context Prompt] -> "{" -> "\n" -> " " -> "risk" -> ":" -> " " -> "HIGH" -> ...(Requires 150 to 500 sequential forward passes) +---> [Field 1: "risk_level"] -------> Logit Slicing -> Top Choice |[Context Prefix Prefill] -+---> [Field 2: "requires_review"] ---> Logit Slicing -> Top Choice(Single KV-Cache State) | +---> [Field M: "action_tier"] ------> Logit Slicing -> Top Choice (All fields evaluated simultaneously)git clone https://github.com/your-org/parallel-constrained-decoding.gitcd parallel-constrained-decoding
python3 -m venv .venvsource .venv/bin/activatepip install -r requirements.txtfrom core.schema import StructuredSchema, FieldDefinition
# Option A: Dictionary-based definitionschema_dict = { "priority": { "type": "enum", "choices": ["P0_CRITICAL", "P1_HIGH", "P2_NORMAL", "P3_LOW"], "description": "Urgency tier based on customer business impact" }, "requires_escalation": { "type": "boolean", "description": "Whether an on-call engineer must be notified immediately" }, "department": { "type": "enum", "choices": ["BILLING", "INFRASTRUCTURE", "SECURITY", "PRODUCT_SUPPORT"], "description": "Target handling department" }}
schema = StructuredSchema(schema_dict)fields = { "tariff_classification": FieldDefinition( name="tariff_classification", field_type="enum", description="Harmonized System 6-digit tariff category code", choices=["0101.21", "0101.29", "8471.30", "8517.12", "8542.31", ...] # Up to 255 choices )}from core.engine import run_parallel_generation
context = """Incident Report: Production database db-primary-01 CPU at 100%.Payment gateway failing for 40% of checkout requests.Tier 1 Enterprise customer affected: Acme Global."""
result = run_parallel_generation(context, schema)
print(f"Latency: {result['elapsed_ms']} ms")print(f"Prefill Time: {result['prefill_ms']} ms")print(f"Passes: {result['sequential_forward_passes']}")print("\nExtracted JSON:")print(result["parsed_json"]){ "mode": "parallel_constrained_calibrated", "elapsed_ms": 74.5, "prefill_ms": 52.1, "suffix_eval_ms": 18.2, "sequential_forward_passes": 1, "is_valid_json": True, "schema_match": True, "parsed_json": { "priority": { "value": "P0_CRITICAL", "prob": 0.9924 }, "requires_escalation": { "value": "true", "prob": 0.9981 }, "department": { "value": "INFRASTRUCTURE", "prob": 0.9815 } }, "field_telemetry": { "priority": { "value": "P0_CRITICAL", "confidence": 0.9924, "cardinality": 4, "top_choices": [ { "choice": "P0_CRITICAL", "probability": 0.9924 }, { "choice": "P1_HIGH", "probability": 0.0068 }, { "choice": "P2_NORMAL", "probability": 0.0006 }, { "choice": "P3_LOW", "probability": 0.0002 } ] } }}from core.engine import stream_naive_generation
for event in stream_naive_generation(context, schema): if event["type"] == "token": print(event["token"], end="", flush=True) elif event["type"] == "done": print(f"\nCompleted in {event['result']['elapsed_ms']} ms")bash run.shpython3 -m uvicorn server.app:app --host 0.0.0.0 --port 8000python3 -m core.benchmark======================================================================Parallel Constrained vs. Autoregressive Generation Benchmark======================================================================--> Running preset: Fintech Fraud Detection (4 fields)... Autoregressive Baseline : 421.3 ms | 148 tokens (122.4 tok/s) | Passes: 148 Parallel Constrained : 74.8 ms | 0 tokens (O(1)) | Passes: 1 >> SPEEDUP: 5.6x faster (Step reduction: 148.0x) >> Schema match: Naive=True | Parallel=True (100% guaranteed)------------------------------------------------------------------------> Running preset: Support Triage Matrix (28 fields)... Autoregressive Baseline : 1894.2 ms | 312 tokens (131.2 tok/s) | Passes: 312 Parallel Constrained : 268.4 ms | 0 tokens (O(1)) | Passes: 1 >> SPEEDUP: 7.1x faster (Step reduction: 312.0x) >> Schema match: Naive=True | Parallel=True (100% guaranteed)------------------------------------------------------------------------> Running preset: High-Cardinality Tariff (1 field, 255 choices)... Autoregressive Baseline : 498.7 ms | 42 tokens (116.5 tok/s) | Passes: 42 Parallel Constrained : 88.6 ms | 0 tokens (O(1)) | Passes: 1 >> SPEEDUP: 5.6x faster (Step reduction: 42.0x) >> Schema match: Naive=True | Parallel=True (100% guaranteed)----------------------------------------------------------------------.├── core/│ ├── __init__.py # SDK package exports│ ├── engine.py # Parallel constrained decoding & autoregressive engines│ ├── schema.py # Schema definitions, metadata compiler & logit mapping│ ├── prompt_builder.py # Prompt templates for prefill catalog and naive baseline│ └── benchmark.py # Command-line benchmark runner├── presets/│ ├── fintech_fraud.json # Fraud detection scenario (4 fields)│ ├── code_security.json # Vulnerability audit scenario (4 fields)│ ├── support_triage.json # Enterprise ticket triage (28 fields)│ └── high_cardinality_255.json # 255-choice tariff classifier├── server/│ ├── app.py # FastAPI endpoints (/api/run-parallel, /api/stream-naive)│ └── main.py # Server launcher├── web/│ ├── index.html # Side-by-side comparison UI│ ├── app.js # Frontend streaming & synchronized scrolling│ └── style.css # UI styling├── MODEL_CARD.md # Hugging Face model card documentation├── requirements.txt # Python package requirements├── run.sh # Startup script└── README.md # Project documentation