Scaling QA at Apple
How I retooled an organization of 200 people at Apple from manual QA to automation engineering, and the first test, which caught search going down.
How do you get 200 people who test by hand to write automated tests?
You don't turn them into engineers. When Apple bought Beats, I joined an organization of 200 people doing manual QA. They knew the product deeply, but they weren't software engineers. So I built a framework that reads plain sentences about the product, and what they already knew became the test. One test could cross web, API, mobile and desktop.
How do you trust a test that has never caught anything?
The first automation we shipped had to be the hardest one. They asked for it first, to prove the approach was worth it. I wrote it myself: one test that followed an app through the whole App Store supply chain. Coding and compiling on desktop, submitting and approving on the web, then searching, buying, downloading and opening on iOS. It ran every five minutes, the finest the schedule allowed.
A few months in, the tool was proven, and monitors showed its alerts. Then one went red, on searching. It had never failed before, so the team and I checked it again. Still red. We raised it with the team that owned the service. They hadn't seen any issues, and the alert was set aside. We checked once more, and raised it again.
Then the news picked it up: a data center was on fire, with a film crew overhead. Once the fire was on TV, the team saw they were down. Search was down, and when you can't search the store, you mostly can't buy. Without the test, the outage would have been found by the manual checkers, by user reports, or by the fire department.
What changed after
Until that day, nobody really knew whether it worked. It had never been flaky, so there was no reason to doubt it, but it had never gone off either. After that, it was taken seriously. The effort was upgraded and became one arm of Platform Infrastructure, a new team at the time.
Who it was for
The manual QAs were worried, the way many people worry about AI now. Automation looked like a risk to them, and like being asked to learn new skills and prove themselves again. That is uncomfortable, and the way through it is to learn what works in the new way.
I didn't teach 200 people at once. I took the ones who were eager and trained them, and empowered them to teach others. Once people saw the tools increased their coverage and reduced the late nights and repetitive testing, they swarmed on the tool. The organization moved from manual QA to automation engineering, and outage detection went from more than 45 minutes to less than 5.
What it costs
A test that has never gone off has to earn trust, and until it does, an alert can be set aside. Five minutes was also the floor: the tests ran on a five-minute schedule, so nothing could be caught faster than that.
The framework is one version of the test harness I've rebuilt across my career, and the service that sent its outage alerts became inline-email.
Want the details?
Ask the window anything this page skips.