Needle 2: What a 14MB Tool-Calling Model Changes About On-Device Agents

The interesting number in Needle 2 is not 45 million. It is 14 megabytes.

Cactus Compute’s Needle 2 is a tool-calling model aimed at phones, wearables, robots, and other small devices. The repository describes a 45M-parameter model, but the weights are compressed to a CQ2-bit artifact that fits inside a 14MB binary. A full session is reported to stay around 28MB of RAM. That is small enough to make the deployment boundary look different from the usual “put a language model behind an API” architecture.

The model does not try to be a general chatbot. Its contract is narrower: declare tools, accept a query, return a structured function call, execute the function, and feed the result back. The API documentation calls this “text in, JSON out,” with a byte-level grammar compiled from the declared schemas. If a request cannot be served by a declared tool, the model returns an empty call instead of improvising free text.

That combination — tiny artifact, bounded memory, schema-constrained decoding, and explicit confidence — is more useful to examine than the headline star count. It tells us what an edge agent can be when the application defines the actions in advance.

The deployment boundary moved

A normal hosted agent starts with an important advantage: the model can be large, its context can grow, and the service can use whatever accelerator is available. The cost arrives every time the application crosses a network boundary. The request has to leave the device, carry potentially sensitive context, wait on a remote scheduler, and return a result. The response may need another round trip when the model selects a tool.

Needle 2 takes a different route. The package is installed once, and the inference engine is cached. After that, the inference path does not need a network connection. The repository documents three ways to provision an air-gapped device: download the engine on a connected machine, copy the cached file into the expected cache path, or set NEEDLE_LIB_PATH to a specific libneedle.so. A Python package downloaded with pip download can also be installed offline with pip install --no-index.

The distinction matters for local agents. A device that controls a thermostat does not need a general-purpose model to discuss poetry. It needs to identify a small set of allowed actions, enforce the argument types, and refuse to execute when the request does not fit. If the first call is a model call, the rest can be ordinary application code. That split is not only cheaper; it creates a place to put authorization, logging, retries, and a manual approval step.

The repository’s quickstart is deliberately plain:

import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a given city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])

The function signature supplies the argument types and the docstring supplies the tool description. Needle returns the executed tool results rather than a free-form explanation. That is a small but important change in responsibility. The model chooses a call; the application decides whether the call is safe to run.

A grammar is an execution policy

Most tool-calling demos stop at JSON output. Needle goes further by compiling constraints into the decoding grammar. A function such as send_money can specify that amount is greater than zero and no more than 10,000, that the recipient matches a pattern, and that the memo is at most 80 characters. The generated call can then be passed to application code without a second parser inventing a permissive interpretation.

The API exposes the same idea for structured extraction. A Pydantic model is effectively a tool with one record shape. Given text, needle.extract() returns a typed object. The README’s example asks for an invoice’s vendor, total, and due date. A record with a missing value is handled according to the contract rather than filled in from the model’s prior.

This is the part I would trust before trusting the model’s prose. Structured output is not a cosmetic property of an agent. It is the boundary between a generated suggestion and a value that can be deserialized, validated, logged, and possibly acted on. A model that emits syntactically valid JSON but invents a room, recipient, or amount has not solved tool calling. It has made the malformed value easier to pipe into a database.

The constraint language has a practical limitation that is worth keeping visible. Literal can restrict a choice, and Field can express ranges, patterns, lengths, and item counts. Those are useful checks on a function call, but they are not a complete authorization policy. temperature being an integer does not prove the user is allowed to set it to 21. amount being positive does not prove the account has sufficient funds. The grammar can reject invalid shapes; it cannot understand whether the operation belongs to the caller.

That boundary is familiar from systems such as Outlines, which constrains generation with regular expressions and context-free grammars, and from Instructor, which structures model output and validates it through Pydantic. Needle packages the same general concern into a much smaller device-facing runtime. The model’s value is not that it solves policy, but that it leaves policy enforcement in a predictable execution layer.

Tool retrieval changes the context budget

The README says that five or fewer tools are rendered directly. Above that, Needle uses a built-in contrastive head to embed the tool schemas, embeds the current query, and selects the top five. The grammar is rebuilt for that subset. An unselected tool is unreachable, not merely less likely.

That is a practical answer to a problem that appears whenever an agent grows from a demo into a product. A large tool catalog sounds flexible until every tool description, required argument, and constraint is placed in the prompt. Context windows are not the only scarce resource; the model’s attention and the application’s latency are also limited. A retrieval step does not magically remove the catalog, but it does make the operating point explicit: retrieve a small candidate set, then constrain generation to those candidates.

The implementation also persists tool embeddings in tool_index_path, keyed by a fingerprint over the schemas and model. A matching fingerprint loads the index immediately; a changed schema re-embeds only what changed. That is a small piece of systems engineering, but it is the kind of detail that makes an offline runtime repeatable. Recomputing an index on every boot would add latency and make the result depend on an unnecessary network service.

There is an engineering trade-off here. Retrieval is useful when the tool catalog is large and the user request usually names a small subset. It can be a mistake when the user’s request is ambiguous, when a tool depends on another selected tool, or when the selected candidate set omits the one function needed to complete the task. Needle exposes the selection mechanism, but the application still has to decide how to handle “none of the above,” escalation, and multi-step calls.

Confidence is a routing signal, not a truth certificate

Needle returns a confidence score for each response. The API documentation says the score is the minimum of two signals: a calibrated post-hoc head over the full prompt and generated call, plus the decoding probability of the call tokens. The proposed product pattern is straightforward: act when the score is above a threshold, and re-ask or route to a larger model when it is below.

This is exactly the sort of feature that looks like a safety control until somebody reads the calibration claim too broadly. A confidence value is not a guarantee that a function will have the intended effect. It estimates something about the model’s call under a particular prompt and model version. A model can be highly confident that it selected the right named tool while the tool’s implementation is wrong, the user lacks permission, or the surrounding state has changed.

The documentation is unusually candid about one boundary: calibration holds for the base model, but fine-tuning does not update the confidence head. A tuned agent reports confidence=None and warns once. That means a deployment that fine-tunes Needle for a different tool catalog cannot quietly inherit the base model’s calibration claim. The confidence threshold must be re-evaluated with the tuned weights and the actual tool schemas.

That limitation suggests a deployment design:

result = agent.complete("turn the kitchen lights to 10%")
if result["type"] == "call":
    if result["confidence"] is not None and result["confidence"] < 0.8:
        ask_for_confirmation()
    else:
        execute_with_authorization(result["function_calls"])

The threshold is not a universal constant. It is a policy parameter that has to be measured against false accepts, false escalations, latency, and the cost of a wrong action. For a read-only request, a low score might be acceptable. For opening a door or moving money, a low score should stop the path. For a model that reports no calibrated score, the correct response is to define the fallback rather than pretending the number exists.

The connection to agent architecture is direct. A multi-stage system does not need every stage to use the same model. A tiny local model can handle high-volume, well-scoped requests, while a remote model handles ambiguous cases. The important artifact is the handoff: a typed call, its confidence, and enough context for the larger model to reproduce the decision. Without that, “use a small model first” is just a promise to do two inferences.

The model is small; the engineering surface is not

Needle 2’s architecture is described as a Simple Attention Network, or SAN. The repository points to A Controlled Study of Attention-Only Transformers, which compares attention-only decoder models with standard transformers while matching parameters, training FLOPs, or depth. The paper reports a gap of 0.006 nats, or 0.27 percent of loss, for one matched-parameter comparison, while noting that attention-only models are weaker where knowledge has to be stored in weights.

That finding is easy to misread. Needle’s tool-calling job is not the same as open-domain factual recall. It needs to map language to a finite schema, follow argument constraints, and keep a bounded session. Those are precisely the places where an on-device model can be specialized. The paper’s reported deficit on knowledge-dense web text does not automatically become a production defect when the application has deliberately removed open-domain answering from the model’s job.

The 256-token sliding window is another sign that the model’s job is being kept honest. Tool descriptions are pinned as KV sinks, and the repository says total memory stays near 28MB no matter how long the conversation runs. A longer context would be convenient, but it would undermine the reason to use a tiny runtime. The model can only work with what fits that window, so long transcripts need a separate application state representation or an explicit reset.

The project also supports LoRA fine-tuning. A tuned checkpoint is merged, quantized, and exported as a .cact file that runs on the same engine. The documented workflow uses needle generate-data, needle finetune, and needle build, with the base model optionally downloaded from Hugging Face. A tuned model remains a single artifact, which is useful for fleet deployment: ship the same runtime and replace the small weights file rather than rebuilding an inference stack for each customer.

But this is where the “14MB model” label needs care. A .cact file can be small, but the surrounding system still needs schemas, embeddings, application code, authorization, network dependencies, and a way to update the model. The operational unit is not only the binary. It is the binary plus the tool contract plus the executor.

That is also why the package metadata lists Apache-2.0 for the Python project even though the GitHub repository metadata reports an MIT license. The repository’s LICENSE is MIT; pyproject.toml says Apache-2.0. I would not build a licensing decision from the package metadata without checking the project’s current distribution and terms. The more useful engineering fact is that the project publishes source and instructions for local inference, fine-tuning, and export under an open license, while the engine artifact itself may have a separate distribution path.

What I would test before putting it on a device

The repository gives enough to form a test plan, but not enough to accept the marketing-sized numbers. I would start with the actual tool schemas and measure four things separately.

First, invocation accuracy. Give the model a fixed set of natural-language requests and count correct tool names, correct arguments, and calls that should have been refused. Keep unsupported requests in the evaluation set. An empty call is a correct result when the tool catalogue is empty; it is a failure if the user explicitly selected a tool that was retrieved incorrectly.

Second, constraint violations. Inject impossible temperatures, negative amounts, malformed handles, extra fields, and values outside the declared ranges. Check whether the grammar rejects them at decode time and whether the executor independently validates them. The second check is not optional.

Third, confidence behavior. Sweep the threshold over a representative set of valid, invalid, ambiguous, and out-of-scope calls. Plot accepted accuracy against escalation rate. The threshold should be chosen from that curve, not copied from a README. Test the tuned model separately, because the confidence head is not updated by the documented LoRA path.

Fourth, memory and latency under the intended interaction pattern. Measure prefill, decoding, peak RAM, and tool-retrieval time with the real number of schemas. The README’s 28MB and 14MB figures describe a particular artifact and engine, not a guarantee for arbitrary prompts, schemas, or host platforms. A mobile application may care more about tail latency than a peak throughput number.

Finally, test the failure path on the device. Stop the network, corrupt a cached artifact, remove a tool, pass an empty catalogue, and feed a result back in the wrong shape. The agent should fail closed for actions and produce a useful escalation signal for reads. An offline model is not valuable if the application silently turns its bounded error into an unconstrained action.

Needle 2 is a useful counterweight to the usual assumption that a useful agent must begin with a large remote model. It does not make a phone equivalent to a hosted frontier model, and its confidence values do not authorize a call. What it does offer is a concrete boundary: small local inference for a defined tool language, with structured execution and an explicit handoff when the model is uncertain.

I am left with the open question that matters for edge agents: which operations are genuinely safe to make local, and which still need the context, policy engine, or broader knowledge of a remote system? The 14MB artifact answers the size part. The rest is still application design.