Needle 2: A 14MB Model That Calls Tools on Any Device

Needle 2: A 14MB Model That Calls Tools on Any Device

Most of the conversation around tool-calling models assumes you need a cloud API or at least a decent GPU. Needle 2, released by Cactus Compute and open-sourced on GitHub, makes a different argument: a 45M-parameter model that fits in a 14MB binary, runs a full session in 28MB of RAM, and ships to macOS, Linux, Windows, Android, iOS, and WebAssembly. No GPU. No cloud. No problem.

What It Actually Is

Needle 2 is a small language model purpose-built for tool calling — not a general chatbot crammed into a tiny form factor. It uses what Cactus Compute calls a Simple Attention Network: a Hadamard MLP replacing the standard feed-forward network, compressed to CQ2-bit (repo; design and ablations: paper).

That distinction matters. Post-hoc quantization typically degrades a model and costs engineering effort to claw back lost accuracy; Cactus's recipe compresses to CQ2-bit as part of the design instead.

The result is a package you can install with pip and call tools from (see the repo quickstart):

pip install cactus-needle

import needle

@needle.tool
def set_lights(room: str):
    "Turn off the lights in a room."
    return {"room": room, "state": "off"}

agent = needle.Needle(tools=[set_lights])
print(agent.run("Turn off the living room lights")["results"])

The Benchmark Numbers

Cactus reports Needle 2 trading wins with larger small models (FunctionGemma 270M, LFM2.5 230M, Apple FM) at a fraction of the size (repo). Per-task scores move with harness and checkpoint, so check the paper and repo for the current tables rather than treating any single vendor number as fixed. The efficiency story is architectural: tens of millions of parameters doing work that takes competitors hundreds of millions.

How It Stays So Small

A few design decisions keep the footprint minimal. The Hadamard MLP is the most obvious one — replacing a standard feed-forward layer with structured linear transforms cuts parameter count without the accuracy cliff you would expect from naive pruning.

Then there is the context management. Needle 2 uses a 256-token sliding window with the tools pinned as KV sinks. The tools stay anchored in the KV cache regardless of how far the conversation drifts, while the sliding window keeps memory bounded during multi-turn sessions. This is a practical engineering choice: you can keep the active tool definitions resident without paying the quadratic attention cost on the full conversation history.

The output is constrained to JSON schemas via a byte-level grammar. This is not an afterthought — it is baked into the decoder so every token the model emits is guaranteed to be valid JSON. If you have ever debugged a tool-calling model that emits malformed JSON with a missing bracket or an unescaped quote, you know why that matters.

Speed on Real Hardware

Theoretical efficiency is one thing. Actual throughput on devices you would actually deploy to is another. Cactus targets phones, wearables, smart-home hardware, and robots (repo) — the kinds of devices where you would actually want on-device tool calling: a screenless smart speaker, a VR headset that needs to adjust settings without latency, a phone running completely offline in a rural area. Published per-device throughput figures were not independently confirmed for this article, so treat vendor speed claims as vendor-reported and measure on your own hardware.

For context on fast tokenization at the edge, our earlier piece on Gigatoken and the Rust modernization wave covers how tokenizers themselves became a bottleneck in LLM pipelines. Needle 2 sidesteps much of this by shipping weights baked into a single 14MB engine with no separate model files to manage (repo).

When the Model Does Not Know

One of the more practical features is the confidence scoring system. Needle 2 emits a confidence score alongside every tool call. When the score drops below a threshold, the system can escalate to a cloud model — a hybrid approach that gives you the latency benefits of on-device inference with the safety net of a larger model when things get ambiguous.

This is exactly the kind of architecture that matters in production. You do not want your smart home controller confidently calling the wrong function because a 45M-parameter model hallucinated a tool name. The confidence score gives you a clean escalation path — one reason vendor benchmark tables deserve a skeptical read before you plan around them.

The Pattern Worth Copying: Confidence-Gated Escalation

Generalize the mechanism above into a routing rule with three tiers. On-device handles the routine: high-confidence calls execute locally, fast and free. The cloud takes the ambiguous: below-threshold scores escalate to a larger model, exactly the hybrid the article describes. A human takes the safety-critical: irreversible actions wait for confirmation regardless of score. Contrast that with "one big model does everything," where every tier pays cloud latency and cost. The tiers use nothing beyond the smart-home-controller example and the article's own framing — but stated as a rule, the pattern ports to any edge deployment, not just this model.

It also maps well to the broader pattern we discussed in agentic AI and scientific computing — the shift from "the model does everything" to "the model does what it can, and the system decides when to bring in something bigger." Verification and orchestration stay upstream. The small model handles the routine cases quickly, cheaply, and locally.

Failure Modes a 45M Tool-Caller Invites

Smallness has its own failure shapes, all derivable from the design above. The 256-token window means tool definitions crowd out conversation history — add too many tools and the model forgets what you asked. The JSON grammar guarantees syntax, not semantics: a wrong-but-valid tool call passes every check the decoder performs. And the confidence threshold cuts both ways — set it high and everything escalates to the cloud (defeating the point); set it low and miscalibrated confidence executes bad calls locally. None of these are reasons to dismiss the model. They are the acceptance tests for deploying it.

Use Cases That Actually Make Sense

The combination of tiny size, JSON-constrained output, and confidence-based escalation opens up some interesting deployment targets:

Voice-to-action on screenless devices. A smart speaker that hears "turn off the kitchen lights at 10 PM" and directly invokes the right tool without round-tripping through the cloud. The 28MB RAM footprint means it runs on the speaker's own hardware.

Offline appliance control. Industrial or agricultural settings where cloud connectivity is unreliable. A 14MB binary running on a low-cost board can parse natural language commands and call device APIs locally — though note the tension in the specs: a full 28MB session footprint fits phones and single-board computers, not the smallest microcontrollers.

Receipt and document extraction. The JSON grammar constraint makes it a natural fit for structured extraction tasks. Feed it a photo of a receipt via OCR text, get back clean JSON with line items, totals, and vendor information — no post-processing regex needed.

What It Means

Needle 2 is not going to replace GPT-5 or Claude for complex reasoning. That is not the point. The point is that a model small enough to fit on a microcontroller can now reliably call tools, emit structured output, and handle multi-turn conversations — with on-device latency that a cloud round-trip cannot match when the network is the bottleneck.

The project is open-source under Cactus Compute (see the repo for license terms). The binary sizes, the benchmark results, the architecture choices — all of it is transparent. If you are building anything that needs on-device intelligence without the cloud dependency, Needle 2 is worth a weekend of experimentation. At 14MB, the cost of trying is measured in download time, not engineering budget.

Sources

• Cactus Compute: cactus-compute/needle — Needle 2 repository
• arXiv: Needle 2 design and ablations