Testing software with AI, and testing AI itself.https://testenigma.com/en-ushttps://testenigma.com/articles/ai-generated-test-cases/https://testenigma.com/articles/ai-generated-test-cases/Turning a user story into Gherkin scenarios with an LLM sounds like free QA coverage. Ungraded, it's also a great way to ship a test suite that looks thorough and only covers the happy path. A coverage rubric for the generator itself.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/catching-llm-hallucinations-regression-harness/https://testenigma.com/articles/catching-llm-hallucinations-regression-harness/A pytest harness that runs the same prompts across model calls, snapshots the outputs, and flags drift automatically, so a silent hallucination regression doesn't ship unnoticed.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/chaos-engineering-for-agents/https://testenigma.com/articles/chaos-engineering-for-agents/The Testing LangGraph Agents post proved the graph is wired correctly under ideal conditions. Real tools fail: timeouts, malformed responses, outages. Inject those failures on purpose and verify the agent degrades gracefully instead of crashing or looping forever.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/cost-latency-regression-testing/https://testenigma.com/articles/cost-latency-regression-testing/A prompt change or model upgrade can triple your cost per request or blow through a latency SLA, and every other eval in this series would call it a pass. Treat token spend and response time as testable, budgetable numbers.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/evaluating-rag-pipelines/https://testenigma.com/articles/evaluating-rag-pipelines/A RAG system fails in two different places (retrieval and generation) and testing it as one opaque blob tells you nothing about which one broke. Separate metrics, separate tests, and a faithfulness check that catches ungrounded answers.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/exploratory-testing-agent/https://testenigma.com/articles/exploratory-testing-agent/An agent that wanders your app clicking things sounds appealing right up until it reports 'looks fine' having tried the same two buttons fifty times. A minimal exploratory agent, plus the one check that actually verifies it's exploring: does it find a bug you know is there?Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/flaky-test-detection-with-statistics/https://testenigma.com/articles/flaky-test-detection-with-statistics/'This test is flaky' is usually a feeling, not a measurement. A formula that turns rerun history into an objective flakiness score, plus a way to cluster dozens of flaky failures down to the handful of root causes actually behind them.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/golden-dataset-for-llm-evals/https://testenigma.com/articles/golden-dataset-for-llm-evals/Five examples hardcoded in a test file works until it doesn't. How to structure, version, and grow a golden dataset that your evals actually scale on.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/llm-as-judge-eval-pipeline/https://testenigma.com/articles/llm-as-judge-eval-pipeline/Keyword coverage runs out the moment 'correct' becomes a judgment call. A rubric-based judge pipeline with structured output: plus the specific ways judge models lie to you, and how to measure it before you trust one in CI.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/llm-test-pyramid/https://testenigma.com/articles/llm-test-pyramid/The classic unit/integration/e2e pyramid, redrawn for systems with an LLM in the loop, and a CI config that actually gates each layer at the right frequency.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/mutation-testing-meets-ai/https://testenigma.com/articles/mutation-testing-meets-ai/Classic mutation testing injects small bugs and checks whether your test suite catches them. Borrow the same method to benchmark an LLM code reviewer instead, and check its false-positive rate while you're at it.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/red-teaming-prompt-injection-tests/https://testenigma.com/articles/red-teaming-prompt-injection-tests/Prompt injection is the LLM-specific vulnerability class that shows up in almost every real incident. A runnable adversarial test suite that checks your own system for system-prompt leaks and injected-instruction compliance before a user finds them first.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/self-healing-selectors-with-llm/https://testenigma.com/articles/self-healing-selectors-with-llm/A renamed CSS class shouldn't fail your test suite the same way a real regression does, but brittle selectors make sure it does anyway. A Playwright wrapper that asks an LLM for a replacement selector when one breaks, tested deterministically without a real browser.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/testing-langgraph-agents/https://testenigma.com/articles/testing-langgraph-agents/Agents add state, branching, and tool calls on top of everything that already made LLM output hard to test. A LangGraph support agent, and three layers of tests that don't require a live API key for the ones that matter most.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/testing-mcp-tool-definitions/https://testenigma.com/articles/testing-mcp-tool-definitions/An agent only knows what a tool does through its schema and description. If the schema promises something the handler doesn't actually do, the agent finds out mid-conversation: as a crash. Contract tests that catch the gap first.Thu, 23 Jul 2026 00:00:00 GMThttps://testenigma.com/articles/write-your-first-llm-eval/https://testenigma.com/articles/write-your-first-llm-eval/The difference between an eval and a regression test, and a minimal pytest-based eval runner you can point at any LLM call: from exact-match scoring to a first pass at semantic grading.Thu, 23 Jul 2026 00:00:00 GMT