convaiinnovations/laya
text-classificationtransformersVerified checksumslayasystem-onecalibrated-decisionsrlcdclassificationrouting
- License
- apache-2.0
- Size
- 2.2 GB
- Revision
- main
Mirrored from convaiinnovations/laya
All models by convaiinnovations →Laya
Multilingual, non-autoregressive System 1 decision model. Give it a state (text, email, ticket, or JSON) and typed questions; it returns typed answers with mathematically calibrated probabilities in a single forward pass (~33 ms) across 100+ languages. Trained with reinforcement learning against strictly proper scoring rules (RLCD), so reporting honest probabilities is the only way to maximise reward. It never generates text, so there is nothing to parse and nothing to hallucinate.
Installation
Python 3.10 or newer. Optional extras: laya[serve] (HTTP server), laya[mcp] (MCP server), laya[langchain] (LangChain and LangGraph), laya[onnx] (ONNX Runtime), laya[fast] (TileLang GPU fast path). Platform-by-platform setup is in the GitHub README.
Long documents. laya-multilingual reads up to 8,192 tokens with max_len=8192. Measured accuracy and time by document length (benchmark script):
Quickstart
Long documents:
laya-multilingualreads up to 8,192 tokens. It ships with a 1,024-token limit that cuts long documents off, so passmax_len=8192for them:In the table above, 16 to 18 of 20 requests were answered correctly with up to about 4,000 tokens of text before them; beyond that results vary (8 to 17 of 20), so check long-document accuracy on your own data. Short inputs give identical answers with
max_len=8192, and speed follows the input's real length, not the limit: short inputs are unchanged, and a 4,000-token input takes about 1.7 s on an Apple GPU. Name the checkpoint withmodel="multilingual", since long mostly-English text would otherwise route to the English checkpoint.
The same call works in any of 100+ languages. The Router detects the script and language and sends non-English text to laya-multilingual:
Fine-tune for better accuracy
The shipped checkpoints work zero-shot, but fine-tuning on decisions from your own domain is where accuracy jumps. On the typed-decisions benchmark (2,000 decisions across four workflows), the fine-tuned laya-typed-decisions checkpoint scores 0.766 accuracy, against 0.362 for the base English checkpoint on the same decisions.
Fine-tuning notebook: runs the whole loop on Kaggle's free 2x T4 GPUs (build the dataset, train, fit calibration temperatures, evaluate, and push the result to the Hub). Details in the GitHub README.
Documentation
nandhakishorm.github.io/laya: guides for prediction hooks, schema-driven decisions, Docker and LangChain and LangGraph, plus a full API reference.
What's new in laya 0.3.20
The checkpoints themselves are unchanged. pip install -U laya for the latest runtime fixes:
-
Long documents with
max_len=8192onlaya-multilingual: a measured table shows accuracy and time by document length. -
Sturdier fast path. After a CUDA out-of-memory error the fallback to CPU switches the TileLang fast path off first, a single-option
choiceno longer crashes it, and concurrent calls can no longer overwrite each other's CUDA-graph buffers. -
Server and runtime.
laya-servedrains its inference pool on shutdown and returns 401 for a malformed bearer header, andONNXAgentmatchesAgenton empty question sets and long conversation lists.
This repo holds all three checkpoints and is the hub for the family. The English checkpoint is at the repo root; the other two are bundled subfolders, and only the one you request is downloaded:
Quickstart: Route Mode (Recommended)
Laya's built-in Router is the recommended way to use Laya in production. It evaluates any state in any language, automatically detects scripts and languages in sub-milliseconds, and dispatches to the optimal checkpoint in a single forward pass.
Every result carries full routing metadata explaining why the choice was made:
Why Route: The Evidence
On a shared benchmark (17,416 questions, one T4 GPU, identical questions per model):
The English checkpoint collapses on non-Latin scripts (Khmer scores 0.000 accuracy at 0.952 confidence). Because the model stays confident while being wrong, confidence gating cannot save you. Router detects the script in <0.5 ms pure Python before the forward pass.
Supplying your own language detection
If you already run a language-identification model, pass its answer instead of relying on the built-in heuristic. lang_guess takes a language code or a callable, is checked after an explicit lang= and before detection, and a callable that returns None falls through to detection:
Production Preload & Memory
A cold checkpoint build costs seconds; language detection costs microseconds. The lazy default keeps two checkpoints resident (english and multilingual, the only two automatic routing chooses between), so after each language's first load a switch costs detection only. A single-language deployment never builds the second. max_loaded=1 rebuilds on every switch (measured at a 7.4 s median reload on CPU and 10.3 s on T4).
For a server or a demo, preload:
Single-Model Mode (Direct SDK)
If you only need a single checkpoint for a dedicated pipeline:
If
laya.load()hangs:transformersprobes for TensorFlow at import, and when TF is installed its abseil runtime can deadlock model construction. Run withUSE_TF=0.
Self-hosting: Jev-compatible HTTP server
laya-serve exposes the Router on the same POST /v1/systemone request and response shape as TypeSafe Jev, so existing TypeSafe clients work by changing their base URL:
It accepts every question shape the Jev API does (for example criteria as a list), ignores unknown fields, and returns a 422 naming the problem for a malformed question. It binds 0.0.0.0 with no authentication unless LAYA_API_KEY is set, in which case it requires Authorization: Bearer <key>.
Architecture
- Backbone: ModernBERT-large (395M, bidirectional, fully fine-tuned) + a decision head trained from scratch: 2 transformer layers, an option-marker scorer, and an act/escalate head. 421M total. (Multilingual uses mmBERT-base, 22 layers, 256k vocab, 322M total).
- Option markers: Every option is scored at its own
[MASK]token, then softmaxed over that question's options. The answer space is defined at request time, so new schemas need no retraining. - Budget: 512 tokens per question for English (
head_max_len = 192); 1024 tokens for multilingual (head_max_len = 256). - Batching: Every question in a call is answered in one single forward pass.
Training
RLCD (Reinforcement Learning for Calibrated Decisions). The policy reports a distribution; exploration adds zero-mean Gaussian noise to the logits; the reward is a strictly proper scoring rule (log + spherical, plus ranked probability score for ordinal questions). Expected reward is maximised only by reporting honest probabilities. Updates are REINFORCE with a group-mean baseline (GRPO-style). Multi-turn conversations use TD(λ=1.0) over prefix slices.
Benchmarks
Measured on a Tesla T4; every checkpoint answered byte-identical questions in the same run.
Speed
103–332 questions/sec batched on a single T4. For reference, TypeSafe Jev has been independently measured at 236–276 ms p50 (AbdelStark, nibzard), so Laya answers a single question roughly 6–8× faster.
Laya (with routing) vs TypeSafe Jev
Every Laya figure is what Router().predict(...) returns — the checkpoint the router selects for that input. Jev figures are third-party published, never measured here (no TypeSafe API access); sample sizes and prompts differ.
On DAIR Emotion, Jev assigned zero probability to the true label on 16% of examples.
Where Jev leads
- High-cardinality label spaces (>20 options at default settings): On Banking77, Jev scores 0.870 (on 72 labels) while Laya scores 0.425 (on 77 labels at default 256-token head budget). Options share a fixed
head_max_lenbudget (192 tokens on English, 256 on multilingual), so 77 options receive only ~3 to 4 tokens per label, causing text to become indistinguishable. Jev supports up to 255 options out-of-the-box. Whilelaya-multilingualsupports 1,024 context (and up to 8,192 in the encoder) and you can raiseagent.cfg["head_max_len"] = 512at runtime, Jev is currently better suited for 50+ options in a single prompt without tuning. - Soft distribution matching: On typed-decisions, while Laya achieves higher argmax accuracy (0.766 vs 0.727), Jev achieves higher soft accuracy (0.580 vs 0.471) against the teacher's full probability distributions.
- Out-of-the-box raw calibration: Before temperature scaling, the base checkpoint has higher raw ECE (0.213 vs 0.144). Laya achieves its 0.081 ECE after domain temperature fitting.
Full report: BENCHMARKS.md.
typed-decisions, measured on all three checkpoints
400 cases, 2,000 decisions, four workflows — measured here.
The fine-tuned checkpoint clears the teacher ceiling and wins all four workflows: invoice processing 0.804, security incidents 0.766, customer service 0.764, agent-trace observability 0.730. By primitive: noul 0.857, choice 0.733, score 0.723.
The base checkpoints sit below the majority-class baseline here — the capability on this benchmark comes from fine-tuning, which is what the fine-tuning notebook is for.
Honest Limits
-
Base checkpoints are near chance on typed-decisions zero-shot — 0.362 here and 0.352 for multilingual, against a 0.318 random and 0.461 majority-class baseline. The 0.766 belongs to the checkpoint fine-tuned on that benchmark's own training split. Laya is a fast base to specialise, not a zero-shot decision engine.
-
High-cardinality choice questions and token budgets: Sequences split into an option prompt budget (
head_max_len) and the remaining document/state budget (max_len - head_max_len):laya(English) defaults to 512 context (head_max_len = 192, ~320 tokens for state).laya-multilingualandlaya-typed-decisionsdefault to 1,024 context (head_max_len = 256, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE). At default settings, a 77-option question like Banking77 allocates only(256 - 16) // 77≈ 3–4 tokens per label, causing accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:
- Raise
agent.cfg["head_max_len"] = 512andagent.cfg["max_len"] = 1024(or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. - Or split large option sets into a two-step coarse-to-fine hierarchical choice.
-
Ordinal
scorequestions are the weakest primitive (SST-5 0.372). -
noulcan follow its option labels instead of the state, most strongly on this English checkpoint.noulrenders its two options asfalse:/true:, and here that label pair can dominate the answer, returning a confident "no" for clearly positive input (#156). Checknoulanswers on your own data. If they look stuck, ask the same question as a two-optionchoicewith neutral keys and your yes/no wording as the descriptions: -
action.act_probabilitycarries no usable signal yet (#185). It reads 1.0 for almost every input, and its raw logits run against correctness (AUROC 0.30 on 396 labelled decisions). Gate onconfidenceinstead, which reaches an AUROC of 0.77 on the same items. -
Ships over-confident: Refitting one temperature per (question type, option count) moves mean ECE 0.466 → 0.081 (
laya) and 0.314 → 0.106 (laya-multilingual). Do this on your own data before trusting the probabilities. -
English only on root: Use
laya-multilingualfor anything outside English.
Links
- GitHub: https://github.com/NandhaKishorM/laya
- PyPI: https://pypi.org/project/laya/
- Live Demo: https://huggingface.co/spaces/convaiinnovations/laya-demo
- Write-up: Read on Dev.to
Apache 2.0 · Convai Innovations