Judgment Labs
Better Judgment: AI Eval Resources
- How to Eval AI Agents — The 2026 Guidehowtoeval.com
Start-here guide: what to measure, how to label, and how to keep evals honest.
- Your AI Product Needs EvalsHamel Husain
The error-analysis loop behind evals that actually move metrics.
- Thoughts on Evals (why offline evals miss production)Ben Hylak
Offline evals and A/B tests only cover the cases you thought of.
- Workshop (open-source agent debugging)Raindrop
Local debugger that hands your agent's traces to your coding agent and turns them into evals.
- How we ship legal-grade evals for agentsGC.AI
In-house legal AI used by Wayfair, Vercel, and Time; LLM judges score every response, and the coding agent runs the whole benchmark loop.
- A Field Guide to Rapidly Improving AI ProductsHamel Husain
Where teams stall: no error analysis, no data viewer, wrong metrics.
- How we build better agents for 15M learnersSpeak
Catching failures in a conversational tutor serving 15M+ learners across 40+ countries.
- Using LLM-as-a-Judge for EvaluationHamel Husain
Critique shadowing and expert alignment for a judge you can trust.
- How Vercel improves their agentsVercel
Production failures become root-caused fixes and regression evals; running at Vercel, Speak, Clay, and Framer.
- The RAG PlaybookJason Liu
Retrieval as a flywheel: synthetic data, segmented evals, provable wins.
- How we reduced memory issues by 27.8%Tolan
Monitoring found the memory and lore failures their benchmarks missed; prompt fixes dropped failures below 0.1%.
- Low-Hanging Fruit for RAG SearchJason Liu
High-leverage retrieval fixes before you reach for a bigger model.
- Agent Self Diagnostics (agents report their own failures)Ben Hylak
Missing context, loops, and capability gaps reported by the agent itself.
- Towards Automating Eval EngineeringLangChain
An eval-engineering skill that reads your repo and traces, interviews you, and emits runnable eval tasks.
- Getting started with automated evaluations (2024)Braintrust
The 2024 offline-eval playbook — LLM judges, heuristics, comparative evals — written when production monitoring wasn't part of the loop.