Why One Agent Often Beats Many: A Cache-Locality Argument

Floris Fok’s article is for people designing agent workflows that repeat work against a large shared context. Its central question is practical: when does assigning a task to more agents cost more than the parallelism saves? Fok argues that splitting related tasks across roles can break prompt-cache reuse. Each role may have a different instruction prefix and may need context handed to it again. The article uses a simple cost model to make that tradeoff visible.

The mechanism matters. Prompt caching reuses previously processed input when requests share an eligible prefix; changing that prefix can reduce reuse. OpenAI’s prompt-caching guide recommends keeping reusable instructions stable and measuring cached tokens, latency, and total cost. A continuous agent can often carry related work through one growing context, while a plan–act–review or specialist routing workflow may create several cold starts and handoffs. This is an architectural hypothesis to test against a real workload, not a rule that every workflow needs one agent.

Fok’s charts compare a single agent with two-agent, three-agent, and routed specialist designs under simulated bursts of related tasks. They show how cold-prefix processing and context handoffs can outweigh cheaper cache reads in the modeled cases. The article explicitly calls this scenario analysis, not benchmark data. Its assumed cache affinity, context size, and task mix determine the result, so the chart should not be read as a measured cost ratio for a particular model or framework.

To use the guide, start with a representative batch of related tasks. Run it through a single-agent baseline and the proposed split workflow with equivalent tools and quality checks. Record input tokens, cached tokens, handoff tokens, total cost, latency, completion time, and outcome quality. If independent tasks finish materially faster in parallel, or specialists need only small separate contexts, additional agents may earn their overhead. Cache behavior also depends on the serving setup and routing, so measure actual hits instead of assuming that one logical agent always shares one physical cache.

Example use cases

Suggested application: support triage. Compare one agent handling several related tickets for the same customer with separate classifier, resolver, and reviewer roles. Check whether repeating the account context causes cold reads.
Suggested application: code maintenance. Give one agent a batch of changes in the same repository, then compare it with specialists per file. Track repeated repository instructions and the time saved by parallel edits.
Suggested application: document review. Evaluate a single reviewer against routed specialists for a set of contracts sharing one policy manual. Measure cache reuse alongside review accuracy.

Sources

Source image: Chart by Floris Fok, from Why One Agent Often Beats Many: A Cache-Locality Argument.

Why One Agent Often Beats Many: A Cache-Locality Argument.