← Jeremiah Walters

Field notes

When AI Agents Cost More Than the Work They Automate

Orchestration, engineering skills, and knowing when to simplify.

I recently spent some time trying to make an existing engineering workflow more agentic.

The process turned reliability requirements into code patterns, detection rules, and validation results. We already had reusable skills for most of that work. What I wanted to know was whether agents could connect them more effectively.

The first experiment was promising. A Worker gathered evidence. An Evaluator reviewed the conclusions. A human accepted the result.

Then I tried to measure whether that was actually better than using the existing skills directly.

That's where things got interesting.

Across two attempts, the coordinator burned through its entire budget setting up the comparison. Neither arm ever ran. No comparison results. No baseline. No measurable improvement.

Cartoon: an original robot labeled Coordinator shrugs with both pockets turned inside out. A moth flies out of one pocket. A receipt beside it reads: budget 0, arm A not started, arm B not started. COORDINATOR EXPERIMENT RECEIPT SetupALL OF IT Arm Anot started Arm Bnot started Budget left0
Two attempts in, the coordinator had nothing left to spend on the actual experiment.

The agents weren't struggling with the engineering problem. They hadn't reached it yet.

Here are the lessons I'm taking forward.

1. The Coordinator Can Become More Expensive Than the Work

The comparison was supposed to be simple. Arm A: use the existing skills directly. Arm B: add Worker/Evaluator coordination. Same evidence, same permissions, same evaluation criteria.

But before either arm could run, the coordinator had to establish source versions, budgets, permissions, review steps, and handoffs. Each decision introduced more context. Each interaction carried that context forward.

I tried again with a smaller context. Same result.

Cartoon: the same robot gets knocked backward by a hit labeled handoff, and its tokens burst out and scatter like coins in a video game, with labels minus context, minus tokens, minus budget. HANDOFF TTTTTTTTTTTT-context-tokens-budget
Every handoff carried the whole history forward. Each one cost tokens, and the coordinator kept dropping them.

In hindsight, this was a known cost, not a surprise:

Bar chart of token usage relative to a chat interaction: chat 1x, single agent about 4x, multi-agent about 15x. ChatChat: 1× the tokens of a chat interaction1×Single agentSingle agent: ~4× the tokens of a chat interaction~4×Multi-agentMulti-agent: ~15× the tokens of a chat interaction~15×
Token usage relative to a single chat interaction, as reported by Anthropic for its multi-agent research system. [1]

Next time, I'd record inputs once, keep contexts small, run independent tasks independently, and pass artifacts instead of conversation histories. [4]

Lesson: Every agent handoff has a cost. Make sure the coordination is worth it.

2. Before Building Another Agent, Look at What You Already Have

One of the most useful exercises wasn't building an agent. It was taking inventory.

Our repo already had skills for requirements analysis, pattern discovery, rule authoring, validation, code review, and incident investigation.

If a skill already knows how to generate a detection rule, why create another agent for it? If a test runner already validates the result, why ask an LLM to simulate that?

A capability doesn't need to become an agent just because an agent could do it.

So I settled on a simpler model:

What I built

StateBudgetsHandoffsManifestsWorkerEvaluatorCoordinator

What actually worked

Existing skillOptional independent evaluationHuman decisiondashed = only when review adds value
Every box on the left is something to design, run, and pay for. The chain on the right reuses what already existed.

Not the most exciting architecture diagram. But easier to reason about, easier to maintain, and much cheaper to coordinate. It's also what Anthropic recommends: find the simplest solution that works, and add complexity only when it demonstrably improves outcomes. [5]

Lesson: Reuse capabilities before introducing agents. More agents don't automatically mean more value.

3. Sometimes the Best Agent Decision Is Not to Act

One investigation asked whether an outbound dependency had an effective concurrency limit. The agent found a control mechanism, but the evidence couldn't establish how it behaved under real deployment conditions.

It could have overreached. Called it a violation. Recommended a new rule. Suggested a config change.

Instead, it produced a qualified finding and stopped.

That was the correct outcome.

In engineering, knowing when not to proceed is part of completing the task correctly. Abstention isn't always a failure state. Sometimes it's the most responsible output.

Lesson: A trustworthy agent should be able to say, "The evidence doesn't justify a change."

4. Passing Tests Doesn't Mean You've Demonstrated Value

After simplifying, I opened a clean workspace and gave Copilot one task: find a narrow detection gap, improve the rule, add regression fixtures, validate.

No coordinator. No A/B experiment. No additional agents.

Review caught a potential false positive. We tightened the matcher. The full test suite passed. The build passed.

Then we ran the old and new rule versions against real repositories.

Both produced zero findings.

No false positives, but also no real-world case exercising the new branch. The tests proved the implementation behaved as expected. They didn't prove it delivered more coverage.

The scan did reveal something else: a value passed through an intermediate collection before reaching its destination. A syntax-local rule can't follow that. The right move isn't to keep stretching the matcher. It's to ask whether a relationship-aware tool fits better.

Agent-generated code needs the same discipline as human-generated code. I now separate four levels of evidence, and here's how our rule change scored:

  1. 1Execution correctness. Did the workflow run?✓ Passed
  2. 2Technical correctness. Does the output satisfy its specification?✓ Passed: full suite green
  3. 3Representative validity. Does it behave correctly under realistic conditions?? Inconclusive: zero findings either way
  4. 4Outcome value. Did it actually improve something that matters?✕ Not demonstrated
Where the rule change landed on each level. Passing tests cleared the first two rungs and said nothing about the last two.

Google's static-analysis team made the same point years ago. They judge a check by whether developers act on its findings, and refine checks against real code before enabling them. [6]

Lesson: A successful execution isn't necessarily a successful engineering outcome.

5. The Experiment Was Becoming the Project

Looking back, this may have been my biggest mistake.

I set out to make a process more efficient. Then I found myself designing agent responsibilities, handoff contracts, state management, approval checkpoints, budgets, and comparison procedures.

Each addition seemed reasonable. Each solved a problem introduced by the previous layer. Eventually, the experiment had become a system of its own.

The turning point came when I stopped perfecting the comparison and asked for one defensible improvement. The result wasn't revolutionary. But it was tangible. And it didn't require an orchestration framework.

Lesson: Prove the smallest useful workflow before designing the system that coordinates it.

So, Did It Work?

Some parts did. The research workflow produced evidence-backed results and stopped when the evidence ran out. The skill-led approach produced a validated rule improvement.

Other parts didn't. The multi-agent comparison never ran. The rule change showed no real-world gain in our sample. And I never proved coordination beat the existing process.

I wouldn't call it a success. I wouldn't call it wasted effort either.

I'm less interested now in how many agents I can coordinate. I'm more interested in:

The goal of agentic engineering shouldn't be to maximize autonomy. It should be to minimize the total effort required to reach a correct, evidence-backed outcome.

If you've been experimenting with agentic engineering, where have you seen the biggest unexpected costs: orchestration, context management, evaluation, or something else?

References

  1. Anthropic Engineering, How we built our multi-agent research system.
  2. Chroma Research, Context Rot: How Increasing Input Tokens Impacts LLM Performance.
  3. Cemri, Pan, Yang, et al., Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657 (2025).
  4. Anthropic Engineering, Effective context engineering for AI agents.
  5. Anthropic Engineering, Building effective agents.
  6. Sadowski et al., Lessons from Building Static Analysis Tools at Google, Communications of the ACM (2018).