The plain definition
A repeatable system that runs your LLM feature over a fixed dataset of inputs, scores each output against a known-good answer or a defined metric, and reports an aggregate quality number you can track over time. Change something, re-run, compare the score. If the number drops, the change regressed — and you find out before it ships, not from a customer.
That is the whole idea. In normal software you write tests: given this input, assert this exact output. LLMs break that model because the same prompt can produce different words each time, and "correct" is often a judgment about meaning, not a string match. An evaluation harness is how you get testing discipline back — it turns a fuzzy "does this feel good?" into a repeatable, numeric measurement you can defend.
Why eyeballing output doesn't scale
Every AI feature starts the same way: you try a handful of prompts by hand, the outputs look great, and you ship. Then reality arrives. Someone tightens the system prompt to fix one bad case — and silently breaks five others nobody re-checked. A new model version "should be better," but on your data it's worse in ways no one measured. Retrieval gets tuned, and answers that used to be grounded start drifting. None of this shows up in a demo. It shows up in production, as support tickets and lost trust.
The root problem is that manual spot-checking has no memory and no coverage. You can hold maybe ten examples in your head; you cannot re-verify two hundred by hand every time you touch a prompt. So changes go out on vibes, regressions accumulate, and the team becomes afraid to touch the system at all. An evaluation harness removes the fear: because every change is scored against the same cases, you can move fast and know you didn't break anything.
What a strong harness actually contains
A ground-truth dataset
The heart of the harness is a set of real inputs paired with known-correct outputs — ideally drawn from your own history: past support tickets and their resolutions, past documents and their correct extractions, past questions and their verified answers. This is what makes the score meaningful. Synthetic examples test whether the model works in a lab; historical ground truth tests whether it works on the cases your business actually sees.
Scorers that match the task
Different tasks need different metrics. Exact-match or structured-field checks for extraction and classification. Retrieval metrics (did the right passage come back, and in what rank?) for RAG. Semantic-similarity or rubric-based scoring — sometimes using a separate model as a judge, held to a fixed rubric — for open-ended answers. Assertion checks for agents: did it call the right tool, with valid arguments, and refuse the actions it should refuse? A good harness combines several so no single blind spot hides a regression.
Backtesting against history
This is the discipline that separates a real harness from a toy. Backtesting replays a candidate change — a new prompt, a new model, a retrieval tweak — over the historical dataset and compares its aggregate score to the current production version, case by case. You see not just that the average moved, but which specific cases got better and which got worse. That per-case diff is what lets you ship a change knowing exactly what it costs and what it buys.
Repeatability and a tracked number
Finally, the harness has to run the same way every time and record the result. One command, same dataset, same scorers, a score you can put on a dashboard and watch across releases. When the number is tracked over time, a regression becomes a visible dip instead of a surprise — and "we improved quality by 8 points this quarter" becomes a claim you can prove.
Where a harness earns its keep: RAG and agents
Two kinds of AI systems fail quietly, and both are exactly where an evaluation harness pays for itself.
RAG pipelines depend entirely on retrieval quality. If retrieval surfaces the wrong passage, the model faithfully summarizes the wrong thing — a confident, well-written, wrong answer. You cannot catch that by reading a few outputs, because they look fine. A harness that scores retrieval against known-relevant documents catches the regression the moment it happens. (More on this in the deeper guide to stopping AI agents from hallucinating with RAG.)
Tool-using agents fail in ways that are even harder to spot by eye: they call a tool they shouldn't, pass a malformed argument, or take an action that's plausible but wrong. Assertion-based evals over historical cases — did it choose the right action, refuse the dangerous one, produce valid arguments — turn agent reliability from a hope into a measured fact.
Evals are the discipline that separates shipped AI from demos
A demo needs to work once, in front of a friendly audience. Production AI needs to work on the ten-thousandth request, on inputs nobody anticipated, after six months of prompt edits by three different people. The thing that bridges that gap is not a smarter model — it's evaluation discipline. Evals are what let a team change a live AI system with confidence instead of fear, and they are the clearest signal that you're dealing with an engineer who ships to production rather than one who builds impressive prototypes and moves on. (If you're hiring for exactly that, see how to vet a production-AI consultant.)
How Hibiscus builds evaluation harnesses
This is not a reading list — it is how Hibiscus Consulting builds AI. Hibiscus Consulting LLC is the software studio of Blake Burnette, a founder-engineer in Cary, North Carolina who ships production AI end to end: model gateways routing tool-using agents, RAG pipelines, real-time voice agents (including one you can call and talk to right now), and evaluation harnesses that backtest output against historical ground truth. The harness is not an afterthought bolted on before launch — it is how each of those systems is proven safe to change.
The positioning is the point: production AI whose every decision is signed, anchored, and independently auditable. An evaluation harness is the front half of that promise (proving quality before you ship); a signed audit trail is the back half (proving what happened after). Both show up as concrete, inspectable artifacts:
- A tamper-evident audit trail signed with post-quantum cryptography (ML-DSA), so decisions are traceable and verifiable after the fact — a provisional patent has been filed.
- Nectar, a programming language written in Rust that compiles to WebAssembly, with a public compiler backed by 2,500+ tests — the same test-everything discipline that a good eval harness brings to AI.
- A multi-tenant payments and field-ops platform (Stripe, Temporal, Postgres, Rust/Node on GCP).
The operational track record behind that: roughly three years as Director of Engineering at a neobank, leading 17 engineers across 5 products and running $720M in payments at 100% uptime (Stripe Connect multi-tenant, Plaid ACH, Apple/Google Pay, fraud rules, KYC/KYB). Blake owned all commits on the Snap! Spend customer-facing React app and wrote core payments code across the platform. He also built a WebRTC platform for the Emmys in React in five weeks, and a React dashboard on a Rails EHR (Medaxion) — healthcare experience on top of nearly a decade of React and TypeScript.
If you want the delivery details — how a harness gets stood up against your own historical data and wired into your release process — see the eval-and-backtesting service.
Stop shipping AI on vibes
If you can't prove a prompt change or model swap made your AI better instead of worse, you don't have an evaluation harness yet. Let's build one against your real, historical data.
Email blake@hibiscus.buzz