Deep-dives, with runnable code.
Where AI meets QA in practice: regression harnesses, evaluation pipelines, and the occasional argument with a flaky test.
AI Wrote the Test Cases: Now Who Tests the Test Cases?
Turning a user story into Gherkin scenarios with an LLM sounds like free QA coverage. Ungraded, it's also a great way to ship a test suite that looks thorough and only covers the happy path. A coverage rubric for the generator itself.
Catching LLM Hallucinations With a Regression Harness
A pytest harness that runs the same prompts across model calls, snapshots the outputs, and flags drift automatically, so a silent hallucination regression doesn't ship unnoticed.
Chaos Monkey for Agents: Breaking Tool Calls on Purpose
The Testing LangGraph Agents post proved the graph is wired correctly under ideal conditions. Real tools fail: timeouts, malformed responses, outages. Inject those failures on purpose and verify the agent degrades gracefully instead of crashing or looping forever.
The Tokens Are the Budget: Cost and Latency Regression Testing
A prompt change or model upgrade can triple your cost per request or blow through a latency SLA, and every other eval in this series would call it a pass. Treat token spend and response time as testable, budgetable numbers.
Precision, Recall, and Vibes: Evaluating RAG Pipelines Properly
A RAG system fails in two different places (retrieval and generation) and testing it as one opaque blob tells you nothing about which one broke. Separate metrics, separate tests, and a faithfulness check that catches ungrounded answers.
Turn It Loose: Building an Exploratory Testing Agent You Can Actually Trust
An agent that wanders your app clicking things sounds appealing right up until it reports 'looks fine' having tried the same two buttons fifty times. A minimal exploratory agent, plus the one check that actually verifies it's exploring: does it find a bug you know is there?
The Flakiness Score: Finding Your Worst Tests With Statistics
'This test is flaky' is usually a feeling, not a measurement. A formula that turns rerun history into an objective flakiness score, plus a way to cluster dozens of flaky failures down to the handful of root causes actually behind them.
Building a Golden Dataset for LLM Evals
Five examples hardcoded in a test file works until it doesn't. How to structure, version, and grow a golden dataset that your evals actually scale on.
Building an LLM-as-Judge Eval Pipeline (and Why You Shouldn't Trust It Blindly)
Keyword coverage runs out the moment 'correct' becomes a judgment call. A rubric-based judge pipeline with structured output: plus the specific ways judge models lie to you, and how to measure it before you trust one in CI.
The LLM Test Pyramid
The classic unit/integration/e2e pyramid, redrawn for systems with an LLM in the loop, and a CI config that actually gates each layer at the right frequency.
Can Your LLM Catch Its Own Bugs? Mutation Testing Meets AI
Classic mutation testing injects small bugs and checks whether your test suite catches them. Borrow the same method to benchmark an LLM code reviewer instead, and check its false-positive rate while you're at it.
Red-Teaming Your Own Guardrails: A Practical Prompt Injection Test Suite
Prompt injection is the LLM-specific vulnerability class that shows up in almost every real incident. A runnable adversarial test suite that checks your own system for system-prompt leaks and injected-instruction compliance before a user finds them first.
Teaching Your Test Suite to Heal Itself
A renamed CSS class shouldn't fail your test suite the same way a real regression does, but brittle selectors make sure it does anyway. A Playwright wrapper that asks an LLM for a replacement selector when one breaks, tested deterministically without a real browser.
Testing LangGraph Agents
Agents add state, branching, and tool calls on top of everything that already made LLM output hard to test. A LangGraph support agent, and three layers of tests that don't require a live API key for the ones that matter most.
Schema or It Didn't Happen: Testing Your MCP Tool Definitions
An agent only knows what a tool does through its schema and description. If the schema promises something the handler doesn't actually do, the agent finds out mid-conversation: as a crash. Contract tests that catch the gap first.
Write Your First LLM Eval
The difference between an eval and a regression test, and a minimal pytest-based eval runner you can point at any LLM call: from exact-match scoring to a first pass at semantic grading.