I recently spent some time trying to make an existing engineering workflow more agentic.
The process turned reliability requirements into code patterns, detection rules, and validation results. We already had reusable skills for most of that work. What I wanted to know was whether agents could connect them more effectively.
The first experiment was promising. A Worker gathered evidence. An Evaluator reviewed the conclusions. A human accepted the result.
Then I tried to measure whether that was actually better than using the existing skills directly.
That's where things got interesting.
Across two attempts, the coordinator burned through its entire budget setting up the comparison. Neither arm ever ran. No comparison results. No baseline. No measurable improvement.
The agents weren't struggling with the engineering problem. They hadn't reached it yet.
Here are the lessons I'm taking forward.
1. The Coordinator Can Become More Expensive Than the Work
The comparison was supposed to be simple. Arm A: use the existing skills directly. Arm B: add Worker/Evaluator coordination. Same evidence, same permissions, same evaluation criteria.
But before either arm could run, the coordinator had to establish source versions, budgets, permissions, review steps, and handoffs. Each decision introduced more context. Each interaction carried that context forward.
I tried again with a smaller context. Same result.
In hindsight, this was a known cost, not a surprise:
- Multi-agent systems are expensive by design. Anthropic reports they use about 15× the tokens of chat, and are a poor fit when agents share context or depend on each other. [1]
- Long context degrades reasoning. Chroma found output gets less reliable as input grows, even on simple tasks. [2]
- Coordination is where these systems fail. A study of 1,600+ multi-agent traces found failures cluster in design, inter-agent misalignment, and verification. [3]
Next time, I'd record inputs once, keep contexts small, run independent tasks independently, and pass artifacts instead of conversation histories. [4]
Lesson: Every agent handoff has a cost. Make sure the coordination is worth it.
2. Before Building Another Agent, Look at What You Already Have
One of the most useful exercises wasn't building an agent. It was taking inventory.
Our repo already had skills for requirements analysis, pattern discovery, rule authoring, validation, code review, and incident investigation.
If a skill already knows how to generate a detection rule, why create another agent for it? If a test runner already validates the result, why ask an LLM to simulate that?
A capability doesn't need to become an agent just because an agent could do it.
So I settled on a simpler model:
What I built
What actually worked
Not the most exciting architecture diagram. But easier to reason about, easier to maintain, and much cheaper to coordinate. It's also what Anthropic recommends: find the simplest solution that works, and add complexity only when it demonstrably improves outcomes. [5]
Lesson: Reuse capabilities before introducing agents. More agents don't automatically mean more value.
3. Sometimes the Best Agent Decision Is Not to Act
One investigation asked whether an outbound dependency had an effective concurrency limit. The agent found a control mechanism, but the evidence couldn't establish how it behaved under real deployment conditions.
It could have overreached. Called it a violation. Recommended a new rule. Suggested a config change.
Instead, it produced a qualified finding and stopped.
That was the correct outcome.
In engineering, knowing when not to proceed is part of completing the task correctly. Abstention isn't always a failure state. Sometimes it's the most responsible output.
Lesson: A trustworthy agent should be able to say, "The evidence doesn't justify a change."
4. Passing Tests Doesn't Mean You've Demonstrated Value
After simplifying, I opened a clean workspace and gave Copilot one task: find a narrow detection gap, improve the rule, add regression fixtures, validate.
No coordinator. No A/B experiment. No additional agents.
Review caught a potential false positive. We tightened the matcher. The full test suite passed. The build passed.
Then we ran the old and new rule versions against real repositories.
Both produced zero findings.
No false positives, but also no real-world case exercising the new branch. The tests proved the implementation behaved as expected. They didn't prove it delivered more coverage.
The scan did reveal something else: a value passed through an intermediate collection before reaching its destination. A syntax-local rule can't follow that. The right move isn't to keep stretching the matcher. It's to ask whether a relationship-aware tool fits better.
Agent-generated code needs the same discipline as human-generated code. I now separate four levels of evidence, and here's how our rule change scored:
- 1Execution correctness. Did the workflow run?✓ Passed
- 2Technical correctness. Does the output satisfy its specification?✓ Passed: full suite green
- 3Representative validity. Does it behave correctly under realistic conditions?? Inconclusive: zero findings either way
- 4Outcome value. Did it actually improve something that matters?✕ Not demonstrated
Google's static-analysis team made the same point years ago. They judge a check by whether developers act on its findings, and refine checks against real code before enabling them. [6]
Lesson: A successful execution isn't necessarily a successful engineering outcome.
5. The Experiment Was Becoming the Project
Looking back, this may have been my biggest mistake.
I set out to make a process more efficient. Then I found myself designing agent responsibilities, handoff contracts, state management, approval checkpoints, budgets, and comparison procedures.
Each addition seemed reasonable. Each solved a problem introduced by the previous layer. Eventually, the experiment had become a system of its own.
The turning point came when I stopped perfecting the comparison and asked for one defensible improvement. The result wasn't revolutionary. But it was tangible. And it didn't require an orchestration framework.
Lesson: Prove the smallest useful workflow before designing the system that coordinates it.
So, Did It Work?
Some parts did. The research workflow produced evidence-backed results and stopped when the evidence ran out. The skill-led approach produced a validated rule improvement.
Other parts didn't. The multi-agent comparison never ran. The rule change showed no real-world gain in our sample. And I never proved coordination beat the existing process.
I wouldn't call it a success. I wouldn't call it wasted effort either.
I'm less interested now in how many agents I can coordinate. I'm more interested in:
- How much human effort does this actually remove?
- How much coordination overhead does it introduce?
- Can I verify the output?
- Does the additional complexity improve the result?
The goal of agentic engineering shouldn't be to maximize autonomy. It should be to minimize the total effort required to reach a correct, evidence-backed outcome.
If you've been experimenting with agentic engineering, where have you seen the biggest unexpected costs: orchestration, context management, evaluation, or something else?
References
- Anthropic Engineering, How we built our multi-agent research system.
- Chroma Research, Context Rot: How Increasing Input Tokens Impacts LLM Performance.
- Cemri, Pan, Yang, et al., Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657 (2025).
- Anthropic Engineering, Effective context engineering for AI agents.
- Anthropic Engineering, Building effective agents.
- Sadowski et al., Lessons from Building Static Analysis Tools at Google, Communications of the ACM (2018).