TL;DR — A local on-device LLM is, by itself, a token generator. It is not an agent. AgentZero Lite turned a Gemma-4 running on a user's machine into something that can hold a real conversation with another AI (Anthropic's Claude CLI in a neighbouring terminal tab) by stacking five distinct layers on top of inference. This is the field report.
Drop a local model into your app and you get tokens. That is all.
You do not get a thing that "decides what to do," that "uses tools," that "knows whether the other side received the message." Those are not properties of the model. They are properties of the system you build around the model. Treating the LLM as if it were already an agent is the most common mistake — and the one that makes "agentic" features brittle.
So a more useful question than "which model is best?" is "what is the smallest set of layers I have to add on top of an LLM before it earns the word agent?"
In AgentZero Lite the answer turned out to be five.
flowchart BT
L[Local LLM<br/>tokens out]
S[+ Structure<br/>GBNF grammar OR native tool template]
T[+ Tools<br/>5-tool catalog over real ConPTY]
C[+ Concurrency<br/>off-UI-thread reactor actor]
P[+ Protocol<br/>identity-first handshake, peer-name routing]
D[Discipline harness<br/>research before code]
A((Agent))
L --> S --> T --> C --> P --> A
D -.-> A
The rest of this doc is one chapter per layer.
The most expensive bug in agent work is not a logic error. It is building the wrong thing because the design decision was a guess. Local-LLM tool-calling is a domain where you can spend a weekend on the wrong backend before you notice.
The Kakashi harness is a small, structured set of agents (security-guard, build-doctor, code-coach, release-build-pipeline) that each own a slice of the engineering process. Before any AIMODE feature work, the harness produced a single concrete artifact: a research note that decided AIMODE would be dual-backend — Gemma 4 via GBNF grammar enforcement, and Nemotron Nano 8B-v1 via its native chat template — and explained why.
Once the decision was an artifact instead of a guess, both backends could be implemented behind one interface (IAgentToolHost) and switched without re-litigating the choice mid-implementation. Discipline produces leverage; it is not overhead.
A common pattern: "Reply with one JSON object: {tool, args}. Do not add prose."
This works inconsistently. Gemma 4 in particular has no native tool-calling SFT — it will helpfully wrap your JSON in markdown, add a friendly preamble, or omit a quote. Asking nicely fails enough of the time to be unusable in a loop.
The trick is to stop asking. Move the contract from the prompt layer to the sampler layer with a GBNF grammar. The model is then physically incapable of emitting a token sequence that does not match the grammar. It still chooses which tool and which args; it cannot choose to break the schema.
root ::= ws "{" ws "\"tool\"" ws ":" ws toolname ws ","
ws "\"args\"" ws ":" ws args ws "}" ws
toolname ::= "\"list_terminals\"" | "\"read_terminal\""
| "\"send_to_terminal\"" | "\"send_key\""
| "\"wait\"" | "\"done\""
args ::= "{" ws "}" | "{" ws kv (ws "," ws kv)* ws "}"
kv ::= string ws ":" ws value
value ::= string | integer | boolean
Source: Project/ZeroCommon/Llm/Tools/AgentToolGrammar.cs
For models that do have native tool calling (Nemotron via the Llama-3.1 chat template), the grammar isn't needed — the structure already exists in the chat format. The application sees both backends through the same IAgentToolHost interface. The lesson generalises: the prompt is for intent, the grammar (or native channel) is for shape. Conflating them is what makes tool-calling flaky.