Cactus-Compute/needle3
text-generationcactus-needleVerified checksumstool-callingfunction-callingon-deviceedgequantizationwebassembly
- License
- apache-2.0
- Size
- 311 MB
- Revision
- main
Mirrored from Cactus-Compute/needle3
All models by Cactus-Compute →Mirrored from Cactus-Compute/needle3
All models by Cactus-Compute →[Image blocked: Needle]
A foundation model for mobiles, wearables, robots, smart home, automotive and microcontrollers. The whole model is a single 8-29 MB file, and we trade general chat capacity to beat models 10x its size on mobile tool calls and match 2-3x bigger models on extraction.
Needle does three jobs, all of them on the device:
[Image blocked: Needle 3 at a glance]
Needle 3 is a Laddered Simple Attention Network, our small-model recipe: a Monarch Hadamard MLP in place of the FFN, GQA attention with causal conv taps, engram n-gram memory read by gather, and multi-lane hyper-connections, trained so that every depth from 2 to 20 layers is a deployable model. Most of its parameters sit in the engram, so the 121M model does the arithmetic of a 50M one. The weights are compressed to CQ2-bit with Cactus Quants; a byte-level grammar compiled from your schemas constrains every token, and every response carries a calibrated confidence score from a learned head. The architecture diagram is on the release page. The repo holds the 20-layer needle3.cact, the needle3.safetensors checkpoint to fine-tune, and an engine per platform.
Tool calling is exact-match accuracy on the full test splits, extraction is field micro-F1 on the full test splits.
[Image blocked: Needle 3 against baselines on six benchmarks]
The interactive frontier plot, the architecture and the fine-tuning results are at cactuscompute.com/needle.
Try it in the browser at cactuscompute.com/needle; the Python package and the source are on GitHub.
Every turn returns one JSON object with function_calls, the model's reasoning and a calibrated confidence; an off-topic request returns an empty list rather than a guess. The engine and the weights are fetched from this repo once and cached.
needle_embed.Needle was designed to be customised. Its capacity is a ladder, and a subnetwork as small as 2 layers, fine-tuned on one product's tools, runs optimally on devices far smaller than the full model needs. Fine-tuning on DroidCall lifts every subnetwork by 18 to 36 points, and from 4 layers up the tuned subnetwork passes DeepSeek V4 Flash, starting at 29M parameters.
[Image blocked: Every subnetwork before and after fine-tuning on DroidCall and on Mobile Actions]
The Python package fine-tunes with LoRA on the frozen base at the full 20 layers, then needle build [--layers N] merges the adapter, slices any subnetwork from 2 to 20 layers and exports a 4-bit .cact that runs on the same engine. The 2-bit post-training and quantisation behind the shipped model, enriched with Cactus proprietary datasets, run on the Cactus Platform.
Every platform folder in this repo holds an engine under 1 MB that loads needle3.cact at start. needle build --platform <folder> [--layers N] fetches the engine and header and puts the weights beside them at any depth, or download the folder here:
[Image blocked: One engine per platform folder]
Tool schemas share the model context with the system prompt and conversation.
needle_init returns the tokenized static-prefix length on success and fails
when that prefix does not fit the model context. In the C API, call
needle_last_error() after a negative return to get the measured prefix tokens
and context limit. Reduce tool descriptions/schema text, split large catalogues,
or declare more than five tools so Needle can keep only retrieved tools in the
per-turn prefix. At completion time, max_new_tokens also reserves context room,
so larger generation caps leave less space for the current turn and history.
The devices guide covers the runner flags, the C API, the WASI component and air-gapped setup.
Needle 3 is built by the Cactus Compute team. If you use it in your work, please cite:
Reach out on founders@cactuscompute.com for partnerships, collaborations, synergies and deploying Needle in your product.
pip install cactus-needleimport needle
@needle.tooldef get_weather(city: str): "Get the current weather for a city." return {"city": city, "temp_c": 27, "sky": "clear"}
agent = needle.Needle(tools=[get_weather])print(agent.run("what's it like in Lagos right now?")["results"])# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]./needle --model needle3.cact --tools tools.json --prompt "dim the living room to 30"./needle --model needle3.cact --tools tools.json --serve@misc{needle3_2026, title = {Needle: Foundation Tool-Calling Model for Tiny Devices}, author = {Ndubuaku, Henry and Mosoyan, Karen and Mroz, Jakub and Cylich, Noah and Kumar, Satyajit and Sandhu, Parkirat and Shemet, Roman and Lee, Justin H.}, year = {2026}, organization = {Cactus Compute, Inc.}, howpublished = {\url{https://github.com/cactus-compute/needle}}}