Testing

Why I cut tests when an AI can write as many as I want, what each layer of testing on dansabin.com proves, and how the agent itself gets tested.

case study · 2026
~/dansabin.com — chat Ask about this project

Key points

Tests have to earn their place

AI makes tests cheap to write, not cheap to keep.

Unit tests first

I strongly prefer them: they're easier to run and maintain, and they point at root causes.

Real infrastructure next

Every pull request deploys a live copy, and tests run against it.

The agent separately

Its evaluations cost real model tokens, so they don't run on every change.

Why cut tests

AI will write as many tests as you ask for. Left alone, it writes too many: I think the tests are a little overboard, and that seems a common issue with AI. A test suite that restates every feature gets slower and harder to change, without proving any more.

So I hold two reasons for a test to exist: Well, we should remove all tests that don't support either 1) a functional feature we need to hold 2) line coverage we need to hold. And I hold to what a test is for: the test should be aiming to test the code, not codify all features directly into unit tests.

I did this to the API's tests. An audit took them from 1,031 to 772, with 250 dropped and 12 merged. It kept the tests that were hard to replace, such as the end-to-end purge, the seven-rotation identity test and the out-of-order failover test.

Where an AI's tests go An AI writes as many tests as you ask for. Every test passes the cut, which keeps a test only for a functional feature or line coverage we need to hold; in the API's audit that took 1,031 tests to 772, with 250 dropped and 12 merged. The coverage floors hold what stays. Testing then runs closest to the code first: ephemeral checks while I work, unit tests before push, and the live environment on the pull request, through the harness. WHERE AN AI'S TESTS GO an AI writes as many tests as you ask for the cut a functional feature to hold, or line coverage to hold the API's tests 1,031 to 772 every test what earns its place 250 dropped, 12 merged CLOSEST TO THE CODE FIRST Ephemeral checks the new lines of a change work, then deleted Unit tests before push, coverage floors 80, 85 and 60 percent Live environment the harness, on the pull request what stays runs Where an AI's tests go An AI writes as many tests as you ask for. Every test passes the cut, which keeps a test only for a functional feature or line coverage we need to hold; in the API's audit that took 1,031 tests to 772, with 250 dropped and 12 merged. The coverage floors hold what stays. Testing then runs closest to the code first: ephemeral checks while I work, unit tests before push, and the live environment on the pull request, through the harness. WHERE AN AI'S TESTS GO an AI writes as many tests as you ask for the cut a functional feature to hold, or line coverage to hold 250 dropped, 12 merged the API's tests 1,031 to 772 CLOSEST TO THE CODE FIRST Ephemeral checks the new lines work, then deleted Unit tests before push, coverage floors 80, 85 and 60 percent Live environment the harness, on the pull request
An AI writes as many tests as you ask for, so every test passes a cut: it stays only if it holds a functional feature or line coverage. In the API's audit that took 1,031 tests to 772. What stays runs closest to the code first: ephemeral checks while I work, unit tests before push, then the live environment on the pull request.

That number didn't stay down. It climbed rapidly as many new features were added, to about 1,800. It's time for another cleanup pass. Code maintenance is now a chore task, and it's a good candidate for AI agent automations.

Why unit tests first

Unit tests are easier to run and maintain, and they point more correctly at root causes. They can also enforce specific decisions the code has taken, so when a change breaks one, it's clear something was written intentionally.

AI could make higher-level tests easier to write, but they lack the speed and the ability to pinpoint issues. They're also more flaky, because several things that aren't wrong can make them fail. They're tied to observed implementation through details such as class selectors, so product changes that don't break the actual experience can break them, in ways that reduce trust in the tests.

Higher-level tests are still necessary, because unit tests can't test the interactions across all the boundaries between components. Tests should live as close to the code they test as possible. So I always start with unit tests, then mocks and unit tests, and finally integration tests, before going higher to behavior tests and end-to-end tests.

If unit tests are hard to write, it's likely because the code isn't easily testable. Writing code with dependency injection and good separation of concerns lets unit testing come in naturally. Where people have leaned on integration tests, I've usually found poor coding standards as the root cause.

What each layer proves

LayerWhat it provesWhen it runs
Ephemeral checksThe new lines of a change work. A scratch script, deleted before the change is doneWhile I work
Unit testsTemplates and wiring, Lambda logic, interface behavior, the harness and scriptsBefore push, on the pull request and again on production
Live environmentThe deployed system behaves, with HTTP calls and headless ChromeOn the pull request
Smoke and routingProduction answers correctly after a deployOn production
Agent evaluationsThe agent's replies hold upSeparately

I think all tests should run and finish before deploy and integration. So no environment gets updated while tests are failing, and the unit tests run again after a merge, because a branch doesn't have to be up to date with main before it merges.

The infrastructure, API and web suites enforce line coverage floors of 80, 85 and 60 percent, and each measures above 94 on the latest production run. The sizes and the pipeline are on the CI/CD page.

The harness

The tests against a live environment are written in Gherkin and run on my own test harness, so they read the same whether they touch an API or a browser.

Testing the agent

An agent's reply changes from run to run, so one pass proves little. Replies are tested on a fixed set of 56 messages, three runs each, with seven checks in code and six scored by a Sonnet judge. The judge never runs on a visitor's reply. The agent page shows how those numbers shaped the design.

What this site doesn't have

Elsewhere I've run smokes on deploys and canaries on a loop, with multiple pre-prods, load tests, and deploys by region with bake times. At Amazon, that was every step of the way. This site has smoke tests on every deploy, and no canaries or bake times, so a bad change reaches both regions.

I can add canaries and bake times quite easily. For now it's mostly cost, and it's also a lot to do with the product. There are no external customers, and I can test my feature set in pull requests with browser testing. If I had API customers, there'd be more reason for bake times and canaries after a production deploy. Automated rollback ties to the same thing. A bake time plus rollback is a great way to interrupt a massive outage by checking increased error rates after a deploy, but none of my customers would cause that at the moment.

Design decisions and trade-offs

DecisionChosen overWhyTrade-off
Cut tests with no feature or line to holdKeeping everything an AI wroteA smaller suite is faster and easier to changeA cut test could have caught something, so the coverage floors have to hold
Deploy a real environment per pull requestMocks aloneTemplates and mocks can't show the parts work togetherEach new environment takes time and runs real AWS resources
Read-only smoke testsFull tests on productionSafe to run on every deployThey don't run a chat turn or solve the human check
Smoke tests, with no canaries or bake timesThe canaries and bake times I ran at AmazonMostly cost, and there are no external customersA bad change reaches both regions
Agent evaluations kept separateRunning them on every changeThey cost real model tokensA change can reach the agent's behavior without an evaluation

Want the details?

Ask the window anything this page skips.