Most teams hear the same advice: “add evals.”
But when you’re staring at a real LLM app, that advice gets vague fast.
Should your first eval be an integration test? A golden dataset? A CI gate? A dashboard metric? An LLM judge?
Our answer? Write your first eval like a test.
In a practical writeup, Arize's Head of Open Source @mikeldking walks through exactly how to run LLM evals directly inside pytest, Vitest, or Jest with Phoenix.
Here's what he covers:
- how evals differ from ordinary tests
- what
Pro tip: not every check should break the build.
Hard invariants belong in CI. Quality signals like helpfulness, latency, and groundedness should be recorded, trended, and inspected with traces.
That gives you a practical first eval without turning normal model variance into
Agent harnesses are becoming the durable layer of AI coding workflows, according to @aparnadhinak.
The model answers once. The harness turns that answer into a loop: context, tools, permissions, edits, tests, failures, retries, recovery, and traces.
That loop decides how
The difference between an agent that works and one that games you comes down to one habit: a good eval.
✅ Spell out the shortcuts you won't accept
✅ Check that the work actually happened
✅ Try to cheat it yourself first
✅ Test it on real traffic
If you can beat it without
Game was on at our AIE after party yesterday, co-hosted with BAND, @awscloud, @crewAIInc, and @Yugabyte. Great conversations, an exciting World Cup game, good vibes only.
Last day at AIE, stop by our booth! Follow the pink 🩷
Our head of DevRel @seldo absolutely rocked through @aiDotEngineer
Two workshops on day 1. Two talks on day 3. 🚀
“Death of the Code Review” : Traditional code review is getting replaced by automated verification harnesses, but someone still has to review that reviewer.
“How
Rustem Feyzkhanov from @SnorkelAI joined us on the Evals Track to break down how they build repeatable, production-like benchmarks for AI agents by benchmarking configurations, not models.
Three takeaways: every company needs its own benchmark, it has to be as close to
Soumya Gupta and Jai Chopra from @Uber was at the @aiDotEngineer Eval Track to present how they use closed-loop evals for their multimodal food photography enhancement agent.
When their agents started reward hacking, they built a closed feedback loop to maintain recursive
Our co-founder & CPO @aparnadhinak talked about the future of evals on the @aiDotEngineer MainStage keynote!
As agents get more complex, the eval method we use should also be adaptive and dynamic: the eval we need is switching from LLM-as-a-judge to agent-as-a-judge.
Come to
Our co-founder and CEO @jason_lopatecki walked through the full anatomy of a self-improving agent: event evidence (traces and evals), context and skills, and a trigger to decide when it runs.
Coding agents are now the ones consuming telemetry data, identifying issues, and
“Which model is cheapest?” is the wrong question.
The better question according to @seldo: which model is cheapest per successful task?
That changes model selection. /1
Flat-rate AI plans hid this.
You could choose “smartest” and ignore the price-performance table.
But newer usage-based pricing models makes the tradeoff visible.
Before you migrate to a cheaper model, measure cost per successful task on your own workloads. That's the number that tells you whether you're actually saving money.
@seldo explains how:
