this hits hard. 'Production hell' perfectly describes my first month trying to run an autonomous agent. It worked beautifully in demos but fell apart in real conditions. The gap between 'prototype works' and 'runs unattended for weeks' is massive.
My breakthrough came when I stopped adding features and started adding failure handling - retry logic, circuit breakers, alerting thresholds. The boring infrastructure work made the difference, not smarter prompts. Production-ready agents aren't about capability, they're about reliability. My night-shift agent now has better uptime than some of my production services, but it took three months of debugging edge cases to get there. Full story: https://thoughts.jock.pl/p/my-ai-agent-works-night-shifts-builds
This was such a great breakdown of the reality of implementing agents! The reliability issues can make improving performance a real challenge. It also gets especially difficult when you can't accurately diagnose why the agent is failing at certain tasks.
What this really shows is that agents are not a software problem but a systems problem. Reliability, cost, and integration failures are symptoms of trying to automate ambiguity. The teams that ship are the ones who first decide what must remain deterministic.
The deterministic backbone isn’t just an engineering safeguard, it’s a governance boundary. Without it, agents don’t fail gracefully — they fail politically, because no one can explain or own their actions.
At that point, the problem stops being technical and becomes organizational.
Fantastic read as ever Hodman. Thanks so much. I was just wondering what you think the best piece of advice would be for how to independently evaluate AI agents without burning through tokens? Is there a way that does this that is effective, because it would seem that the normal processes end up with a situation where the AI agents basically end up marking each other's homework?
Thank you! The "AI grading AI" problem is real and gets worse as you scale evaluation.
To answer your question, I would recommend checking whether your agent did what it was supposed to do, rather than trying to evaluate how it thought through the problem.
For example, if your agent is supposed to book a meeting, you can check: (1) Did it actually create a calendar event? (2) Is the time correct? (3) Did it succeed? You can verify all of this without asking another AI to judge the work.
For more complex situations, create a collection of real examples with known correct answers, then regularly test your agent against them. Track when it starts getting things wrong. This gives you a reliable baseline without needing AI judges.
I'm planning on writing about how I tackle this with Asaura AI over the next couple of months.
this hits hard. 'Production hell' perfectly describes my first month trying to run an autonomous agent. It worked beautifully in demos but fell apart in real conditions. The gap between 'prototype works' and 'runs unattended for weeks' is massive.
My breakthrough came when I stopped adding features and started adding failure handling - retry logic, circuit breakers, alerting thresholds. The boring infrastructure work made the difference, not smarter prompts. Production-ready agents aren't about capability, they're about reliability. My night-shift agent now has better uptime than some of my production services, but it took three months of debugging edge cases to get there. Full story: https://thoughts.jock.pl/p/my-ai-agent-works-night-shifts-builds
This was such a great breakdown of the reality of implementing agents! The reliability issues can make improving performance a real challenge. It also gets especially difficult when you can't accurately diagnose why the agent is failing at certain tasks.
You can't fix what you can't trace, and stochastic failures make pattern recognition nearly impossible.
Excellent piece Hodman.
Thank you! I have a follow up coming out in the next couple of days!
Looking forward to it. I experienced it on a smaller scale when putting together an infrarag system so the read hit home. Lots of lessons learned.
Here you go! https://www.thedataletter.com/p/ai-agent-starter-kit
What this really shows is that agents are not a software problem but a systems problem. Reliability, cost, and integration failures are symptoms of trying to automate ambiguity. The teams that ship are the ones who first decide what must remain deterministic.
You make a good point! The breakthrough isn't a new model, but the engineering discipline to define the deterministic backbone.
Agreed — but I’d push it one step further.
The deterministic backbone isn’t just an engineering safeguard, it’s a governance boundary. Without it, agents don’t fail gracefully — they fail politically, because no one can explain or own their actions.
At that point, the problem stops being technical and becomes organizational.
Fantastic read as ever Hodman. Thanks so much. I was just wondering what you think the best piece of advice would be for how to independently evaluate AI agents without burning through tokens? Is there a way that does this that is effective, because it would seem that the normal processes end up with a situation where the AI agents basically end up marking each other's homework?
Thank you! The "AI grading AI" problem is real and gets worse as you scale evaluation.
To answer your question, I would recommend checking whether your agent did what it was supposed to do, rather than trying to evaluate how it thought through the problem.
For example, if your agent is supposed to book a meeting, you can check: (1) Did it actually create a calendar event? (2) Is the time correct? (3) Did it succeed? You can verify all of this without asking another AI to judge the work.
For more complex situations, create a collection of real examples with known correct answers, then regularly test your agent against them. Track when it starts getting things wrong. This gives you a reliable baseline without needing AI judges.
I'm planning on writing about how I tackle this with Asaura AI over the next couple of months.
Incredibly comprehensive as always ! Thanks for sharing ✨
My pleasure! Just trying my best!