Testing
Why I cut tests when an AI can write as many as I want, what each layer of testing on dansabin.com proves, and how the agent itself gets tested.
Key points
AI makes tests cheap to write, not cheap to keep.
I strongly prefer them: they're easier to run and maintain, and they point at root causes.
Every pull request deploys a live copy, and tests run against it.
Its evaluations cost real model tokens, so they don't run on every change.
Why cut tests
AI will write as many tests as you ask for. Left alone, it writes too many: I think the tests are a little overboard, and that seems a common issue with AI. A test suite that restates every feature gets slower and harder to change, without proving any more.
So I hold two reasons for a test to exist: Well, we should remove all tests that don't support either 1) a functional feature we need to hold 2) line coverage we need to hold. And I hold to what a test is for: the test should be aiming to test the code, not codify all features directly into unit tests.
I did this to the API's tests. An audit took them from 1,031 to 772, with 250 dropped and 12 merged. It kept the tests that were hard to replace, such as the end-to-end purge, the seven-rotation identity test and the out-of-order failover test.
That number didn't stay down. It climbed rapidly as many new features were added, to about 1,800. It's time for another cleanup pass. Code maintenance is now a chore task, and it's a good candidate for AI agent automations.
Why unit tests first
Unit tests are easier to run and maintain, and they point more correctly at root causes. They can also enforce specific decisions the code has taken, so when a change breaks one, it's clear something was written intentionally.
AI could make higher-level tests easier to write, but they lack the speed and the ability to pinpoint issues. They're also more flaky, because several things that aren't wrong can make them fail. They're tied to observed implementation through details such as class selectors, so product changes that don't break the actual experience can break them, in ways that reduce trust in the tests.
Higher-level tests are still necessary, because unit tests can't test the interactions across all the boundaries between components. Tests should live as close to the code they test as possible. So I always start with unit tests, then mocks and unit tests, and finally integration tests, before going higher to behavior tests and end-to-end tests.
If unit tests are hard to write, it's likely because the code isn't easily testable. Writing code with dependency injection and good separation of concerns lets unit testing come in naturally. Where people have leaned on integration tests, I've usually found poor coding standards as the root cause.
What each layer proves
| Layer | What it proves | When it runs |
|---|---|---|
| Ephemeral checks | The new lines of a change work. A scratch script, deleted before the change is done | While I work |
| Unit tests | Templates and wiring, Lambda logic, interface behavior, the harness and scripts | Before push, on the pull request and again on production |
| Live environment | The deployed system behaves, with HTTP calls and headless Chrome | On the pull request |
| Smoke and routing | Production answers correctly after a deploy | On production |
| Agent evaluations | The agent's replies hold up | Separately |
I think all tests should run and finish before deploy and integration. So no environment gets updated while tests are failing, and the unit tests run again after a merge, because a branch doesn't have to be up to date with main before it merges.
The infrastructure, API and web suites enforce line coverage floors of 80, 85 and 60 percent, and each measures above 94 on the latest production run. The sizes and the pipeline are on the CI/CD page.
The harness
The tests against a live environment are written in Gherkin and run on my own test harness, so they read the same whether they touch an API or a browser.
Testing the agent
An agent's reply changes from run to run, so one pass proves little. Replies are tested on a fixed set of 56 messages, three runs each, with seven checks in code and six scored by a Sonnet judge. The judge never runs on a visitor's reply. The agent page shows how those numbers shaped the design.
What this site doesn't have
Elsewhere I've run smokes on deploys and canaries on a loop, with multiple pre-prods, load tests, and deploys by region with bake times. At Amazon, that was every step of the way. This site has smoke tests on every deploy, and no canaries or bake times, so a bad change reaches both regions.
I can add canaries and bake times quite easily. For now it's mostly cost, and it's also a lot to do with the product. There are no external customers, and I can test my feature set in pull requests with browser testing. If I had API customers, there'd be more reason for bake times and canaries after a production deploy. Automated rollback ties to the same thing. A bake time plus rollback is a great way to interrupt a massive outage by checking increased error rates after a deploy, but none of my customers would cause that at the moment.
Design decisions and trade-offs
| Decision | Chosen over | Why | Trade-off |
|---|---|---|---|
| Cut tests with no feature or line to hold | Keeping everything an AI wrote | A smaller suite is faster and easier to change | A cut test could have caught something, so the coverage floors have to hold |
| Deploy a real environment per pull request | Mocks alone | Templates and mocks can't show the parts work together | Each new environment takes time and runs real AWS resources |
| Read-only smoke tests | Full tests on production | Safe to run on every deploy | They don't run a chat turn or solve the human check |
| Smoke tests, with no canaries or bake times | The canaries and bake times I ran at Amazon | Mostly cost, and there are no external customers | A bad change reaches both regions |
| Agent evaluations kept separate | Running them on every change | They cost real model tokens | A change can reach the agent's behavior without an evaluation |
Want the details?
Ask the window anything this page skips.