Scaling Reliable AI Agents

Kadhir Manickam

(4 minutes)

<section data-section-id='949ae167-9639-4e27-aeda-fd2b66804bfb'>Running an agent once is easy. Running it thousands of times a day is a different sport - an incredibly frustrating problem to overcome when putting agents in production. LLMs are non-deterministic, they do weird stuff...a lot. That's easy to hide in a demo and brutal in production, especially when you're chasing 9s of reliability. Every weird edge eventually shows up at enough volume. And the worst part is that the tail of failures is so wide that two things get hard fast: reproducing an issue, and hillclimbing your way past it. There are too many one-off failures to replay by hand, so the only way to know if you've solved a problem is to run in high volume and watch the probabilities instead. At ProductNow, we run agents at scale with 9s of reliability, but we had to discover three key principles to get us here. They all point the same way: ask the model to do as little as possible, and let deterministic engineering carry the rest. </section> <section data-section-id='57a94627-1e98-426e-91a8-587ddad03ce4'>1) Shrink what the agents do The first principle: put determinism everywhere the model does not need to think. In practice, every part of the system that touches the outside world is a deterministic script. When we fetch information from an external system, a script does it, with a strict schema on its inputs and outputs. When we write back downstream, same thing. That leaves the agent in the middle doing the minimum: it assembles context inside ProductNow and acts on it. A one-off agent can afford to be loose. Ours run thousands of times a day at high reliability, and we've built our systems with that constraint in mind. But that raises a question: what does the agent in the middle actually do? </section> <section data-section-id='2839fded-5736-42ef-974b-44cef34f1ddd'>2) Let the schema drive the loop The answer is a retry loop. Every script in our harness takes a particular schema on its inputs and outputs, and that contract sits at the center of the design. On each call, the agent's job stays small: assemble context and produce a payload that matches the schema. When the payload misses, validation rejects it and hands the error back. The agent adjusts and retries until the output hits the schema. Only then does the script run. <mermaid data-height="260"> flowchart LR A[Agent assembles context] --> B[Agent produces payload] B --> C{Schema validation} C -->|Miss: error returned| B C -->|Pass| D[Deterministic script runs] </mermaid>This is what makes our agent loops so effective. Every pass carries a crisp pass-or-fail signal, retries are cheap and converge fast, and the retry logic lives in the harness, so the model spends its tokens on judgment instead of plumbing. That clean signal powers hillclimbing too. We watch the first-pass hit rate across thousands of runs, and every change to a prompt or a schema shows up in that number. </section> <section data-section-id='e6187445-8f62-4764-a6ae-ccc04aab2d3f'>3) Make search the interface Even with a tight loop, the agent still has to find the right context. That's the second place where reliability leaks out. Point a model at a repository and tell it to open the right files, and it'll generally do the right thing. But every now and then, it'll read the wrong file, follow a stale reference, and burn tokens on paths that go nowhere. But it does it enough to ruin our 9s. So we made search the interface instead. Rather than telling agents which files to open, we tell them which terms to search for and which queries to run. The queries are fixed, and the results they return are baked in, which pushes the determinism into the search layer, where we control it. That shift does two things for us. First, it asks the model to do less on every call. Reading a whole file and judging what matters is expensive and unpredictable. Running a query that returns a handful of relevant snippets is cheap and behaves the same way every time. Second, it gives us a surface we can actually test. Because every lookup is a query, we built a separate set of retrieval evals around the search layer. We know what a good result looks like for each query. We've been running search as an industry for decades now. There's a ton of precedent and techniques to draw on here, so we can easily benchmark what the agent should've received for each query it submitted to achieve optimal results. In other words, we've found that agent reliability goes up when it can repeatedly submit search queries to find the information they need than directly accessing files. Overall, it's a more pointed context for the agent to handle, with more determinism in the loop. This is the balance we're after. The model keeps the flexibility it's good at, judging and acting on what comes back. The system owns the part that has to be repeatable. Deterministic where it counts, flexible where it helps. </section> <section data-section-id='901e3ef9-1d64-4cba-ba36-da9ca0dadc6b'>What this buys Put all of this together and you get 9s of reliability on genuinely complex workflows. You also cut token spend, cut latency, and keep context windows free, because the model carries as little as possible and the engineering carries the rest. The harness itself is the hard part. Wrapping a deterministic shell this tight around a non-deterministic core is a real engineering problem, and solving it is where the reliability lives. If you are fighting the same long tail of weird failures, we'd love to hear what is working for you. </section>