Why you can't ship AI on vibes
Every team that puts an LLM in production hits the same wall. Someone tweaks a prompt, swaps a model, or adjusts the retrieval settings, everyone agrees it "feels better," and it ships. Two weeks later a customer surfaces an answer that used to be right and now is wrong. Nobody can say when it broke, which change broke it, or whether the last fix quietly broke three other things. That is what shipping on vibes gets you: a system that drifts in the dark.
The way out is the same discipline every serious software team already applies to code — a test suite — adapted to the fact that LLM output is fuzzy rather than exact. That test suite is an evaluation harness, and building a good one is a real engineering project, not a spreadsheet of prompts. This is applied production work: it draws on live systems, including a real-time voice agent you can pick up the phone and talk to, and a public compiler carrying 2,500+ tests.
What an evaluation and backtesting harness actually does
Grade output against historical ground truth
The core of the harness is a curated set of real past cases where you already know the correct answer — support tickets that were resolved, documents that were classified, questions that have a settled right answer. The harness replays each case through the current system and scores the output against that ground truth. When you change a prompt, model, or retrieval step, you re-run the whole set and see, case by case, what got better and what got worse. That is backtesting: the change proves itself against history before it touches a live user.
Stop RAG and agent regressions before release
Regressions in AI systems are sneaky because they are usually partial. A retrieval tweak that helps 80% of queries can silently degrade the other 20%, and no amount of eyeballing the demo will surface that. A harness wired into your workflow runs on every proposed change and flags the specific cases that regressed — the exact wrong retrieval, the agent step that now takes a wrong turn — so it never reaches a user. It turns "we think this is safe" into a list of cases you can look at.
Turn "it seems better" into defensible metrics
"Seems better" cannot be reviewed, argued with, or reported to a stakeholder. A harness replaces it with numbers you can defend: accuracy against ground truth, retrieval hit rate, faithfulness (did the answer stick to the retrieved evidence?), regression count, and pass rate on cases that must never fail. Grading fuzzy output takes real design — exact-match, rubric scoring, and LLM-as-judge each fit different problems, and choosing wrong gives you metrics that look precise but mean nothing. Getting that right is most of the job.
Pair every result with a signed, auditable decision
A metric is only as trustworthy as your ability to reproduce it. Hibiscus pairs eval results with signed, anchored records of exactly which inputs, retrieved passages, model version, and prompts produced each score — so a result is independently verifiable rather than something you take on faith. When a regulator, a customer, or your own team asks "how do you know this model is better," the answer is a record anyone can check, not a screenshot of a dashboard.
What a harness engagement looks like
- Assemble the ground-truth set. Mine your real history for cases with known-correct answers, and build the hard-case and must-never-fail subsets that matter most for your domain.
- Design the scoring. Pick the right grader per task — exact-match, rubric, or LLM-as-judge — and calibrate the judge against human labels so the scores mean what you think they mean.
- Wire it into the loop. Make the harness run on every prompt, model, or retrieval change, with a clear pass/fail gate and a regression report that names the cases that moved.
- Anchor the results. Sign each run to its exact inputs and configuration so any result can be reproduced and independently verified later.
This is production work, not slideware
Evaluation is not a side deliverable at Hibiscus — it is how the work is built. Hibiscus Consulting LLC is the software studio of Blake Burnette, a founder-engineer in Cary, North Carolina who ships production AI end to end: model gateways routing tool-using agents, RAG pipelines, evaluation harnesses that backtest against historical ground truth, and real-time voice agents — including one you can call and talk to right now, a far harder bar than a text demo because latency and interruption handling are unforgiving.
The testing discipline behind the harness work is not aspirational. Nectar is a programming language written in Rust that compiles to WebAssembly, with a public compiler backed by 2,500+ tests — the same reflex of proving correctness by exhaustive, repeatable checks that a good eval harness applies to LLM output. The broader positioning is the point:
- Production AI whose every decision is signed, anchored, and independently auditable — the principle that makes eval results verifiable instead of a matter of trust.
- A tamper-evident audit trail signed with post-quantum cryptography (ML-DSA); a provisional patent has been filed.
- A multi-tenant payments and field-ops platform (Stripe, Temporal, Postgres, Rust/Node on GCP).
The operational track record behind that: roughly three years as Director of Engineering at a neobank, leading 17 engineers across 5 products and running $720M in payments at 100% uptime (Stripe Connect multi-tenant, Plaid ACH, Apple/Google Pay, fraud rules, KYC/KYB). Blake owned all commits on the Snap! Spend customer-facing React app and wrote core payments code across the platform. He also built a WebRTC platform for the Emmys in React in five weeks, and a React dashboard on a Rails EHR (Medaxion) — healthcare experience on top of nearly a decade of React and TypeScript.
Add an eval harness to your AI system
If you can't prove your last change made the AI better — or catch the one that made it worse before a user does — let's build the harness that turns guesswork into evidence.
Email blake@hibiscus.buzz