Needle 2: A 14MB Model That Calls Tools on Any Device

Most of the conversation around tool-calling models assumes you need a cloud API or at least a decent GPU. Needle 2, released by Cactus Compute and open-sourced on GitHub, makes a different argument: a 45M-parameter model that fits in a 14MB binary, runs a full session in 28MB of RAM, and ships to macOS, Linux, Windows, Android, iOS, and WebAssembly. No GPU. No cloud. No problem.

What It Actually Is

Needle 2 is a small language model purpose-built for tool calling — not a general chatbot crammed into a tiny form factor. It uses what Cactus Compute calls a Simple Attention Network: 27 layers, 512 wide, with a Hadamard MLP replacing the standard feed-forward network. The whole thing trains on a 115B token corpus with 38B tokens of post-training, and the quantization (CQ2-bit) is baked in from pretraining rather than applied after the fact.

That distinction matters. Post-hoc quantization degrades a model and you spend engineering effort clawing back lost accuracy. Training the quantization in from the start means the model learns to operate within its compressed representation natively.

The result is a binary you can ship to a Raspberry Pi and call tools from:

# Download and run Needle 2
curl -LO https://github.com/cactus-compute/needle-2/releases/latest/download/needle-2
chmod +x needle-2
./needle-2 --system "You control smart home devices." --prompt "Turn off the living room lights"

The Benchmark Numbers

Needle 2 leads both in-domain and out-of-distribution on Seal-Tools, and posts a strong 63.7 on Mobile Actions. Here is how it stacks up against other small tool-calling models:

The MFLOPs column tells the real story. Needle 2 burns 70 MFLOPs per token. LFM2.5, at 230M parameters, burns 460 — more than six times the compute for a model that does not lead it on the benchmarks that matter. FunctionGemma at 270M parameters is at 540 MFLOPs. Raw size does not correlate with capability here; architecture choices do.

How It Stays So Small

A few design decisions keep the footprint minimal. The Hadamard MLP is the most obvious one — replacing a standard feed-forward layer with structured linear transforms cuts parameter count without the accuracy cliff you would expect from naive pruning.

Then there is the context management. Needle 2 uses a 256-token sliding window with KV sinks for the system prompt. The system prompt stays anchored in the KV cache regardless of how far the conversation drifts, while the sliding window keeps memory bounded during multi-turn sessions. This is a practical engineering choice: you can maintain system instructions without paying the quadratic attention cost on the full conversation history.

The output is constrained to JSON schemas via a byte-level grammar. This is not an afterthought — it is baked into the decoder so every token the model emits is guaranteed to be valid JSON. If you have ever debugged a tool-calling model that emits malformed JSON with a missing bracket or an unescaped quote, you know why that matters.

Speed on Real Hardware

Theoretical MFLOPs are one thing. Actual throughput on devices you would actually deploy to is another:

These are not synthetic benchmarks on a server rack. These are the kinds of devices where you would actually want on-device tool calling — a screenless smart speaker, a VR headset that needs to adjust settings without latency, a phone running completely offline in a rural area.

For context on fast tokenization at the edge, our earlier piece on Gigatoken and the Rust modernization wave covers how tokenizers themselves became a bottleneck in LLM pipelines. Needle 2 sidesteps much of this by shipping as a self-contained binary — no external tokenizer dependency, no Python stack, no library versioning headaches.

When the Model Does Not Know

One of the more practical features is the confidence scoring system. Needle 2 emits a confidence score alongside every tool call. When the score drops below a threshold, the system can escalate to a cloud model — a hybrid approach that gives you the latency benefits of on-device inference with the safety net of a larger model when things get ambiguous.

This is exactly the kind of architecture that matters in production. You do not want your smart home controller confidently calling the wrong function because a 45M-parameter model hallucinated a tool name. The confidence score gives you a clean escalation path.

It also maps well to the broader pattern we discussed in agentic AI and scientific computing — the shift from "the model does everything" to "the model does what it can, and the system decides when to bring in something bigger." Verification and orchestration stay upstream. The small model handles the routine cases quickly, cheaply, and locally.

Use Cases That Actually Make Sense

The combination of tiny size, JSON-constrained output, and confidence-based escalation opens up some interesting deployment targets:

Voice-to-action on screenless devices. A smart speaker that hears "turn off the kitchen lights at 10 PM" and directly invokes the right tool without round-tripping through the cloud. The 28MB RAM footprint means it runs on the speaker's own hardware.

Offline appliance control. Industrial or agricultural settings where cloud connectivity is unreliable. A 14MB binary running on a $15 microcontroller can parse natural language commands and call device APIs locally.

Receipt and document extraction. The JSON grammar constraint makes it a natural fit for structured extraction tasks. Feed it a photo of a receipt via OCR text, get back clean JSON with line items, totals, and vendor information — no post-processing regex needed.

What It Means

Needle 2 is not going to replace GPT-5 or Claude for complex reasoning. That is not the point. The point is that a model small enough to fit on a microcontroller can now reliably call tools, emit structured output, and handle multi-turn conversations — and it can do it faster than you can make an API call to a cloud provider.

The project is fully open source under Cactus Compute. The binary sizes, the benchmark results, the architecture choices — all of it is transparent. If you are building anything that needs on-device intelligence without the cloud dependency, Needle 2 is worth a weekend of experimentation. At 14MB, the cost of trying is measured in download time, not engineering budget.