Test harness
A Gherkin test framework I've rebuilt across my career, from Expedia through Beats Music to Apple. This version runs dansabin.com's tests in plain English that a person and an AI can both read.
The harness is something that I've been building time and time again and continues to be useful. Now with AI it's even more valuable. This version is the harness/ package inside the dansabin.com repository, and it runs this site's integration tests. It isn't released yet.
Why write tests in plain English if an AI writes them?
Because the person who reads a failure isn't the one who wrote the test. Plain English isn't critical for writing tests anymore, but it is for understanding where they fail. This one checks an API over HTTP:
Scenario: Every API hostname's health check reports ok
When I GET "https://{{host}}/health" for each "host" in "api_hosts"
Then every response status is 200
And every response JSON at "status" equals "ok"
And every response header "cache-control" equals "no-store"
This site has 23 Features like it, with 48 Scenarios. They run on every pull request that changes code, and the smoke ones run against prod after every deploy.
What happens when a test fails on a page that works?
It started as a warning. Whenever a page redirected during a run, WebdriverIO printed a line in the middle of the test output. The agent I was working with traced it and found it cost nothing: no step failed or waited longer because of it, and the harness was about to move out of this repository into its own package.
The move was why I wanted it fixed. The harness is written for people outside this repository first, and they'd only see the line. If it was going to be its own package, it should get better on the way out, not leave as it was. If we want to wait we should do so in ways that don't look like errors.
So we rebuilt the waits, and to see them hold, the agent ran the harness's browser check 60 times in a row in real Chrome, against a page that reloads itself. Some runs failed, and one of them showed this:
And I can see "#flip"
Error: WebDriver Bidi command "script.callFunction" failed with error: unknown error - Cannot find context with specified id
"A real bug: a Check can fail on a working page, and main has it too." The visibility check ran a small script in the page, and if the page swapped while it ran, Chrome turned it away. We talked through whether we needed our own script at all, and settled on both: WebdriverIO's own checks in place of ours, and a second read whenever the page changed under a read. Both went into the same change.
The number to watch: failures in 60 runs
What we watch now is a loop: the harness's own browser check, run 60 times in a row in real Chrome against a small local site, and how many of those runs fail. On a pull request, a failure like that is a required check gone red on a page that works, and a rerun before the change can merge. Each rerun adds to how long a change takes to merge, and makes that time harder to predict, so a loop that gets this number to 0 keeps the time to merge tight. The loop itself is small: one run takes about 17 seconds on a laptop, with nothing deployed, so 60 runs is about a quarter of an hour.
| 60-run loop | Runs that failed |
|---|---|
| Before the final fixes | 9 |
| Again, before the final fixes | 6 |
| With the final fixes, on a page that reloads 3 times and then stays | 0 |
The last loop also swapped the test page: the old one reloaded every 160 ms and never stopped, so even with the fix a check could run out of time on it, and no visitor could have used that page either. The loop's pages are built to be harsh, so it finds a rare failure fast; it doesn't say how often a real page would hit one.
What it is
A testing framework that lets authors write integration tests that are easier to operate and maintain. It can still test any user facing feature, and it can manipulate environment and state, so a test stays on the case at hand instead of going through a brittle setup path that has nothing to do with it.
A repository supplies a configuration and its Features, and writes no Step code. Features are behavior written in Gherkin, which a person and an AI can both read, and they run deterministically in CI. Each Step reads its input straight from between the quotes in the Gherkin, so Steps stay one to one with what the Feature says, and it waits while the tab's page is being replaced before it acts.
One language
The point is that a test reads the same whatever it touches. The Scenario at the top of this page checks an API over HTTP. This one drives a browser, against a mock API that can be told to fail:
Scenario: A Visitor_Limit refusal says when sending works again and returns the message with Send
When I open "{{web_url}}/mock-api/mode/visitor-limit"
And I open "{{web_url}}/"
And I type "What have you built lately?" into "[aria-label='Your message']"
And I click ".message-box-send"
Then the text of ".chat-banners-refusal" contains "You can send again after"
And there are 0 ".chat-tab-visitor*=What have you built lately?"
And the field ".conversation-view-panel:not([hidden]) [aria-label='Your message']" holds "What have you built lately?"
And the text of ".conversation-view-panel:not([hidden]) .message-box-send" contains "Send"
And there are 0 "button*=Resend"
And there are 0 "button*=Try again"
Why WebDriver
Today the harness makes HTTP requests and drives headless Chrome. It has no iOS or desktop driver yet.
I chose WebDriver because that's where those surfaces can come in. It's a standard protocol, and drivers such as Appium speak it for iOS and desktop apps, which is why this uses it instead of Playwright. I could put Playwright under the Gherkin if I wanted, but for now it's only WebdriverIO. The next step is a check of dansabin.com on iOS, after launch.
Where it comes from
I've built this more than once, over the length of my career.
- At Expedia, I worked with the QA lead to develop a scalable WebDriver farm that automated web and iOS testing.
- At Beats Music, my work was creating a testing framework around the web applications using Ruby, Cucumber and Selenium which was responsible for testing the application during all stages of development from PRs through to production against all supported browsers.
- At Apple, which bought Beats, the work became a framework that allowed tests to span Web, API, Mobile, and Desktop systems in a single context.
- This version I started for this site, because I couldn't find anything similar.
The best story is from the Apple system, not this version. One process tested the whole life of an app through the App Store: coding and compiling on desktop, submitting and approving on the web, then searching, buying, downloading and opening on iOS. It caught an outage and pinpointed the one service responsible in that supply chain, within 5 minutes of the incident starting.
How it's built
The Steps and parallel Workers in a Run share one document store in memory, NeDB, so a Step can capture a value and a later Step can use it. Scenarios run in parallel and stay apart through keys that include the Scenario's id.
A few rules hold the rest together. Every Step goes on the record, even when it passes, because a passing test that sent the wrong request is bad too. A test that can't honor what it was asked to do fails rather than guessing. The Features have no fixed sleeps; the one timed pause is a 50 ms settle inside the scroll Step. And a test fails if anything under harness/ names this site, so it can be pulled out later.
Where this is going
This is what I'm working on, and it isn't built yet. The plan is one vocabulary, one runner, one report, with the Gherkin never knowing which backend runs it. The headline is one Scenario that crosses them: create an order through the API, see it appear in the browser, approve it on the phone, request the API and expect the status approved. The store is what would carry the order's id from one step to the next. No Feature crosses them like that today, though the Apple system did.
The phone is the sharpest reason. The things that break on a phone are the ones a resized window cannot show, such as the iOS keyboard hiding the submit button. So the first device work drives the web app in mobile Safari on an iOS simulator and Chrome on an Android emulator. Native apps and desktop apps come after that.
How I use it here
I test mechanics with unit tests and features with the harness. The Features cover the static site, the chat frame, the seeded chats, and the site at phone widths in Chrome. The same Features run against a deployed environment, the local dev server and a mock API, chosen by config and tag.
Outcome
It's still a work in progress. It does catch things: if you break part of the flow, it's able to detect that. Buttons that stop working, a bug in how a form works, how a page is supposed to load and how scroll works are the usual catches.
The false failures came early, because integration tests have to wait for things. Getting those waits right, without failing too soon or retrying badly, took real work.
I'm its only user today, and a repository can't add its own Steps yet.
It's designed to be open source, but needs a little more testing across the rest of the devices before that happens.
Want the details?
Ask the window anything this page skips.