Plenty of engineering teams say AI is making them faster. Then the work runs into the same bottlenecks: Code written in an afternoon waits days for review. Agents get built and then fenced in, because nobody trusts them enough to let them act. The tools have sped up, but the process around them has stayed where it was.

At Afresh we’ve tried to take a more balanced approach. There’s no dogmatic AI-first agenda here, nor is anyone dismissive or cynical about AI. Everyone is thinking about it very carefully and intelligently. That comes through in the way we approach tasks. We’re critical about where AI can make an impact, and then actually let it help.

We build our agentic capabilities ourselves, using state-of-the-art methods to keep up with the frontier of agent design. This means we’re constantly validating what works for us and what doesn’t, whether it’s a new model or a change to the system prompt. A big part of that is how we check our work, and how quickly we can check it decides how fast we can move. For us, that comes down to evals.

Evals are ordinary. Grocery isn’t.

Every team that builds agents runs evals, and the basic machinery of ours looks like everyone else’s. The real work is in the answer key. In grocery, that work starts before you can write a single expected answer, because “correct” depends on knowledge the data doesn’t hand you.

Take shrink: product shipped into a store that never sold because it spoiled, was stolen, or was otherwise lost. Say an agent needs last month’s strawberry shrink for one store. The data has a field that looks like the answer: scanouts. When store associates throw product away, they scan it out first, and that scan becomes the record. Summing scanouts produces a clean, specific number—and that number is wrong. Most shrink never gets recorded. Product walks out the door, gets mis-scanned at the register, gets damaged in transit, or gets tossed without a scan. In many grocery departments, shrink runs as high as 10–15% of sales, and scanouts catch only part of it.

The traps keep going. Some of those strawberries may have been cut into fruit cups for the deli, so they left produce without being sold or wasted. An inventory count of “1” might mean one clamshell or one case. A count might have been taken at open or at close. None of these errors look like errors. The answer comes back confident and plausible, which is what makes it dangerous.

Then there’s the question of whose definition. Some of what an answer key needs is industry knowledge: what shrink is, how product moves from a distribution center to a store, what a department manager does with a count. Some of it is specific to the business. Two grocers can define shrink differently, and a company with several banners, separate store brands under one parent, may not track it the same way across all of them. The right answer depends on who’s asking.

And none of this is trivia. Shrink shapes what a store believes it has on hand, which drives what it orders. Undercount it and the store thinks it has more strawberries than it does, orders too few, and shoppers find an empty display. Get it wrong the other way and the extra product spoils. In a low-margin business selling food that lasts mere days, both mistakes cost money, and the second one becomes food waste.

“The quality of the system is directly tied to the quality of the eval set.”

This is where our domain knowledge earns its keep. An answer key written by someone who knows where scanouts fall short rejects the clean but wrong answer. One written without that knowledge passes it, and every score built on top of it measures the wrong thing very precisely. Because we know grocery this well, we can write evals we actually trust.

That knowledge also has to live somewhere a system can find it and use it. Some of it is defined metrics in our data warehouse, so agents query governed metrics instead of improvising their own math. Some of it is glossaries and ontologies that capture what grocery terms mean and how they relate to each other. Some of it is models trained on grocery-specific data.

All this knowledge goes to both our evals and an agent’s own knowledge base. That means our evals can hold an agent to the standard a grocer would. An agent, in turn, gets what it needs—the metrics, terms, and relationships it reasons with—so it can meet that standard.

But feeding the same knowledge to both risks an agent learning the test instead of the domain. So how you give a system knowledge matters as much as the knowledge itself. The “how” is familiar territory for us: Afresh has been doing machine learning since its founding to forecast sales and predict stores’ inventory needs, using the same math and science that sits under today’s language models. That means we’re well-versed in the techniques that can give an agent knowledge to reason with instead of memorizing the answers, so it holds up on the cases we tested and the ones we didn’t think to include.

Judges scale the knowledge

Exact-match checks handle answers that reduce to a number. Most of what our agents return doesn’t. It’s a paragraph that might reach the right conclusion in an unhelpful form, or the wrong conclusion in a very convincing one. For those, we use LLM judges, and we’re starting to explore a newer class of fast decision models.

We run judges for correctness and separate ones for how well an answer communicates. Does “twenty” match “20”? Is the right number buried under three sentences of preamble? Most pet peeves people have about AI answers can become a judge criterion, which turns a complaint into a measurement. The criteria are also where grocery knowledge shows up again, because a judge only knows what a good answer looks like if someone who knows grocery guided it.

Judges run live, across hundreds of comparisons that would take a person days or weeks to review. We won’t claim we’ve done anything novel with them yet. It’s ongoing work: we add and refine judges to make them cheaper and faster as technology improves and our understanding of the problem space grows. What they give us is reach. With our judges, the knowledge in the answer key gets applied to every answer, every time we run the set.

How it makes our teams faster

When a new model comes out, advertised as cheaper or more effective, we point it at the eval set and find out whether it gets the right numbers from our tools more often or less. When someone has an idea for restructuring the context an agent starts with, they run it and see whether it ticks more boxes or fewer, instead of asking twenty questions and trusting their gut. We don’t need to write new tests for every change, though we keep adding evals as we learn. Ideas that might otherwise be risky bets become quick experiments.

The resulting confidence is what let us drop code review as a default. Engineers still ask for a review whenever they want a second set of eyes, and security tests run alongside the quality evals. This is how we work while prototyping and iterating, and our prototypes are built to a higher fidelity than the word usually suggests, so it isn’t a license to be loose with critical data. The better our evals get, the more of the checking they can carry.

The result is what the go-faster-with-AI pitch usually promises and rarely delivers. Engineers here spend their time building, trying things, and finding out quickly whether they worked, instead of waiting in a queue. The AI does real work, the evals tell us when it gets something wrong, and the people deciding what to trust know what the evals cover and what they don’t.

If you’d like to build that way, on problems where getting the answer right actually matters, take a look at our open roles.