Stop putting every tool in your agent's context

Published June 21, 2026

When you give an LLM agent a tool, you are "binding" it. Binding means the tool's full definition (its name, its description, and the shape of its arguments) gets added to the prompt the model reads, on every single turn of the conversation, even if the model never uses that tool.

Bind ten or twenty tools and this is no problem. Bind a few hundred, and you have quietly used up a large chunk of the model's context window (the total amount of text it can read at once) before the conversation has even started.

This post walks through a small demo project built to make that cost visible, and to try a fix: instead of binding every tool, bind just two, search_tools and invoke_tool, and let the agent look up what it needs, only when it needs it.

Why this matters

Nobody sits down and designs a thousand-tool agent on purpose. It happens gradually: an integration for Jira here, one for Slack there, an internal API wrapped as a dozen more tools. None of these feel like a big decision on their own. But add them up over a few months and a team can end up binding far more tools than anyone planned for. This is not hypothetical: Anthropic has reported a setup with five MCP servers (a standard way of connecting external tools to an agent) spending about 55,000 tokens on tool definitions alone. Tokens are the chunks of text a model reads and is billed for, so that is a real cost paid before any actual work happens.

Static binding (binding every tool, all the time) doesn't fail with an error message when this happens. It fails quietly, in two ways:

• Cost: every tool definition is a fixed tax paid on every turn, whether or not that turn touches the tool.
• Accuracy: as the library grows, tools with overlapping names or descriptions get confused for one another more often. The model has more near-identical options to choose between, and picks the wrong one more often.

Both problems get worse as the library grows, not as actual usage grows. A tool that gets called once a day still rides along on every single message sent that day. That mismatch, paying for size instead of paying for use, is the actual problem this project tries to solve.

The setup

The repo starts with twenty hand-written local tools: a calculator, some string utilities, a hashing tool, base64 encoding, basic statistics, that kind of thing. Each one is a real, callable LangChain @tool, not a fake stand-in, so the numbers in this post come from actual tool schemas, not estimates.

Twenty tools wasn't enough to make the difference between approaches obvious, so the tool library was grown programmatically: one real conversion tool for every (category, from_unit, to_unit) combination across thirteen categories, including length, mass, volume, digital storage, duration, frequency, and torque. That adds up to roughly 2,000 tools, each one independently callable, each with its own name and description.

Measuring the cost

Using the real OpenAI-style function-calling schema (convert_to_openai_tool) and tiktoken (a library that counts tokens the same way OpenAI's models do) for an honest token count:

	Tools bound	Tokens	% of a 131k context window
Static (bind everything)	~2,038	~97,000	~74%
Dynamic (search_tools + invoke_tool)	2	~142	~0.1%

Binding everything eats nearly three-quarters of gpt-oss:latest's real context window, before the system prompt, the conversation, or even a single tool result has been added. Binding just the two meta-tools costs almost nothing in comparison.

Tradeoff: static binding is simple. There's no extra lookup step, no moving parts, and every tool the model could use is right there in front of it from turn one. That simplicity is exactly why it's the default. It only stops being a good default once the tool count grows large enough that the token cost (and the accuracy problem) starts to outweigh the convenience.

Search isn't free either

Replacing static binding with search_tools → invoke_tool trades a context problem for a latency problem (latency just means delay): every tool call now needs an extra round trip. The model has to call search_tools first, read the results, and only then call invoke_tool. That's two model turns instead of one.

Tradeoff: this approach saves a huge amount of context, but every tool use now takes longer and costs an extra model call. For a tool the agent uses constantly, paying that extra round trip on every single call is wasteful. Search is a good fit for the long tail of rarely-used tools, and a bad fit for the small set of tools used over and over.

Fixing the new problem: a usage cache

The fix is a small usage cache (tools/cache.py). Every time invoke_tool successfully calls a real tool, it increments a counter for that tool's name, and saves the counter to disk so it survives restarts. Once a tool's counter crosses a threshold, it gets "promoted": from then on, it's bound directly to the agent (and the system prompt is told it's "already available"), so the model can call it straight away instead of searching for it first. The agent rebuilds its list of bound tools before every turn, so a tool that just became popular is available immediately, with no restart needed.

Tradeoff: this adds real complexity, a persisted counter, a promotion threshold, a cap on how many tools can be promoted at once, in exchange for getting the best of both earlier approaches: most tools stay out of context until needed, and the few tools used often skip the search step entirely. The threshold and cap are tuning knobs: promote too eagerly and you're back to bloating the context; promote too conservatively and frequently-used tools keep paying the round-trip tax.

The result is a system that starts cheap (two meta-tools) and adapts over time: the handful of tools actually used often enough end up bound directly, while the other ~99% stay out of context until genuinely needed. This is the same idea behind defer_loading in Anthropic's own tool search tool: keep the long tail of rarely-used tools out of the prompt, let the model look them up through a cheap search step, and let real usage, not a guess made at design time, decide which tools earn a permanent spot.

How it fits together

flowchart TB
    DEF["definitions.py<br/>ALL_TOOLS (~2,000 tools)"]
    ING["ingestion.py<br/>build_tool_index"]
    VS[("Vector store<br/>name + description embeddings")]
    CACHE[("cache.py<br/>ToolUsageCache (persisted)")]
    PROMOTE["promote_hot_tools"]

    subgraph AGENT["main.py: agent loop"]
        MODEL["ChatOllama"]
        BOUND["Bound tools (retrieval.py):<br/>search_tools, invoke_tool,<br/>+ promoted hot tools"]
        MODEL --- BOUND
    end

    DEF --> ING --> VS
    BOUND -- "search_tools(query)" --> VS
    VS -- "top-k matches" --> BOUND
    BOUND -- "invoke_tool(name, args)" --> DEF
    DEF -- "result" --> BOUND
    BOUND -- "record_use(name)" --> CACHE
    CACHE -- "hot_tool_names()" --> PROMOTE
    PROMOTE -- "bind directly,<br/>skip search next turn" --> BOUND

All 2,000 real tools live in definitions.py. Once, at startup, ingestion.py turns each tool's name and description into an embedding (a list of numbers that captures what the text means) and stores it in a vector store, so similar tools can be found by meaning, not just by matching keywords. None of those 2,000 tools are bound to the model directly.

What is bound is search_tools and invoke_tool, the two meta-tools defined in retrieval.py. When the agent calls search_tools, it searches the vector store and returns the few best-matching tools. When the agent calls invoke_tool, it runs the real tool and reports the result back, and also tells ToolUsageCache that this tool was used. Before every turn, promote_hot_tools checks the cache and hands back the small list of tools that have been used often enough to deserve direct binding. That closes the loop: what the agent did in past turns changes what it doesn't have to search for next turn.

A request, end to end

It's easier to see with one concrete request. Say the user asks to convert 5 miles to kilometers:

1. The agent calls search_tools("convert miles to kilometers") and gets back the closest match, convert_length_mi_to_km, by meaning rather than exact keywords.
2. It calls invoke_tool("convert_length_mi_to_km", {"value": 5}), gets the answer, and the cache quietly records one use of that tool.
3. Ask for the same conversion a few more times and the counter crosses the threshold. Now convert_length_mi_to_km is bound directly, and the next request answers in a single turn, no search needed.

The first call costs an extra round trip. By the fourth, the tool has earned its place in context and behaves exactly like a statically bound tool would, while the ~2,000 conversion tools nobody asked for stay out of the prompt entirely.

When do you actually need this?

If your agent has a small, fixed set of tools, and binding all of them still leaves most of the context window free for the actual conversation, static binding is almost always the right call. It's simpler to build, easier to debug, and the token cost is small enough not to matter.

This pattern is worth reaching for once one or more of these is true:

• Your tool count is in the hundreds or thousands, often because tools come from many integrations (MCP servers, internal APIs) added over time rather than designed as one set.
• Tool definitions are eating a noticeable share of your context window, leaving less room for the actual conversation.
• You're seeing the model pick the wrong tool because several tools have similar names or descriptions.
• Most of your tools are used rarely, while a small handful are used constantly. That is exactly the shape that makes a usage-based promotion cache worth its complexity.

If none of that describes your agent yet, static binding is fine. This project exists for the point where it stops being fine.

Watching it work

The project ships a Streamlit demo (src/agent-tool-search/app.py) that puts all of this on screen at once:

• The real token cost of static vs. dynamic binding, measured against the model's actual context window (queried live from Ollama, not assumed).
• Which tools are currently promoted to direct binding, and why: a one-line explanation that searching-then-invoking costs latency the cache is designed to avoid.
• A live trace of the agent's tool-picking process, streamed turn by turn, including the model's own reasoning content where the backend exposes it (reasoning=True on ChatOllama surfaces this directly, no prompt engineering required).

What could be improved

None of these are dealbreakers, and not all of them are even about search in particular, a few are just the next problems you run into once an agent has access to a large toolbox. They are worth naming because they point at where this goes next:

• Search is only as good as the descriptions. A tool is found by embedding its name and description, so a vague or misleading description means the right tool never surfaces for the right query. Moving the cost out of the context window puts a new cost on writing clear tool descriptions.
• Search can hide the detail a tool needs to be called correctly. search_tools returns only a name and a one-line description, not the parameter schema. So the model doesn't see, for example, that a date argument must be ISO 8601 (2026-06-21) and not June 21, or that an amount is expected in cents rather than dollars, until it tries to invoke and gets it wrong. For tools whose correct use depends on a specific input format, keeping the full schema and a couple of example values bound up front is worth the extra tokens. Accuracy beats compactness there, so those tools are good candidates to bind directly rather than hide behind search.
• The cache never forgets. Usage counts only ever go up; there is no decay or eviction. A tool used heavily for one afternoon stays "hot" until five others outrank it, even if it is never touched again. That is fine for a demo, but a real system would want counts to age out so promotions track recent usage, not all-time totals.
• Tool results can flood the context just as fast as definitions. Keeping definitions out of the prompt buys little if invoking a tool then dumps a 10,000-row API response straight back into it. The same discipline applies on the way out: a tool should return only the fields the model actually needs, summarized or paginated, not the full raw payload. This demo's tools return short strings, so it never hits this, but anything wrapping a chatty API has to decide what is worth the model's context and what is just noise.
• Some work is better written as code than repeated as tool calls. If a task means invoking the same tool hundreds of times, say, finding the top users by token usage in every sub-department in Eng above some threshold, search-and-invoke once per call is the wrong shape entirely. It is far cheaper and more reliable to have the agent write a small program that does the loop and run it once. The fix there isn't smarter tool retrieval; it's giving the model a way to execute code.
• Executing generated code needs a sandbox. The moment you let an agent write and run a program, you have handed it arbitrary code execution. That has to run locked down, no network, no filesystem access beyond a scratch space, with strict time and memory limits, or a helpful agent becomes a remote-code-execution hole. This demo sidesteps the problem by only ever invoking the fixed, hand-written tools in definitions.py, but anything that takes the previous point seriously has to solve this first.

Each of these is worth a post of its own, and I plan to dig into them in upcoming write-ups: schema-aware search, trimming bulky tool results, code execution instead of repeated tool calls, and sandboxing. Stay tuned.

What this isn't

This is a demo, not a recommendation to bind two tools and call it done for every agent. The real takeaway is narrower: the cost of static tool binding scales with the size of your tool library, and once that library gets large enough, a search-based layer plus a usage-driven promotion cache is a much cheaper way to keep an agent capable without permanently taxing every single request for it.

The full source, including the tool-generation code, the caching layer, and the Streamlit demo, is in the agent-tool-search repo.