Publishing on time at Prime Video

How the effort I led at Prime Video took on-time publishing of the highest value content from 80% to 99.5%, and why solving each issue wasn't enough.

case study · 2020 to 2025
~/dansabin.com — chat Ask about this work

How did launches go from 80% on time to 99.5%?

For Prime Video's highest value content, on time means live within 90 seconds of launch. The effort I led, across two teams of engineers, took that from 80% to 99.5% by checking every title against its due date all the time, and telling the team that owned a problem early, in working hours, what was blocking it.

Why wasn't fixing each missed launch enough?

In my first year, a title was going out with 20 pieces of artwork. The limit was 19, so it failed a check. An email went out asking someone to pick one to drop. The email was never acted on. The title missed its launch, and tens of thousands of customers noticed at once. The launch date had been known months before.

That was how launches were missed then. Launch checks generally weren't done. Emails went back and forth between operators and studios, through customer support. Most of the misses were preventable. Operators started checks by hand three days before launch, but three days wasn't always enough, and some titles had less than that.

Most failures were missing assets, or assets that failed because they were incomplete, such as missing artwork. Harder were local-language assets. Some regions required local-language audio or subtitles as a quality gate, and a content provider who didn't know would miss the launch in that territory.

Checking every title against its due date

The supply chain kept failing the same way, publishing deadline missed, because titles failed and nobody knew. We built an index of every title and its due date, two pieces of data that didn't live next to each other before, so any failure could be judged by what it was blocking. A single important or urgent failure was enough to trip an alarm. The usual alarms still fired only above expected thresholds, because most issues were data quality and some failures were expected.

To keep the index current, we built a near-real-time mapping system that kept titles and dates in sync as contracts and title metadata streamed in. The mapping was challenging because we had to map a dozen ids across as many services, and do it as the data changed constantly, to keep the mapping accurate. It was hardest because Prime Video isn't one catalog; it's a catalog of everyone's catalogs. The same movie is available through a dozen or more providers, but customers don't want to see duplicates. They want a single movie with multiple ways to watch. Tracking each of these through the ids, their merges and the updating contracts was the first challenge. Then came the scale: tens of millions of products with millions of updates a day, on a supply chain checking this data at thousands of requests per second across hundreds of services.

A title's last days before launch, before and now Before, the launch date was known months before, checks by hand started three days before launch, and emails went between operators and studios, and the launch was missed. Now titles and dates are kept in sync, a single failure that matters trips an alarm, the team that owns it is told in working hours, and the launch is on time. BEFORE launch date known months before three days before checks by hand emails operators and studios launch missed NOW titles and dates kept in sync a single failure trips an alarm the team that owns it told in working hours launch on time A title's last days before launch, before and now Before, the launch date was known months before, checks by hand started three days before launch, and emails went between operators and studios, and the launch was missed. Now titles and dates are kept in sync, a single failure that matters trips an alarm, the team that owns it is told in working hours, and the launch is on time. BEFORE launch date known months before three days before checks by hand emails operators and studios launch missed NOW titles and dates kept in sync a single failure trips an alarm the team that owns it told in working hours launch on time
Before, the launch date was known months before, checks by hand started three days before launch, and emails went between operators and studios, and the launch was missed. Now titles and dates are kept in sync, a single failure that matters trips an alarm, the team that owns it is told in working hours, and the launch is on time.

Supply chain teams could call a service with their errors and get back a severity, so they could prioritize failures before they became high-severity title blockers. High-risk problems reached them in working hours, not as pages in the middle of the night when a launch was already missed or at risk. Services could also call before scheduling work, to learn the priority of what to work on, which effectively turned the supply chain into a distributed priority queue. The priorities had to agree, hence a single service vending them. Otherwise what one service rushed, others wouldn't, and the misses would continue.

One severity service for the whole supply chain Supply chain teams call with their errors, and services call before scheduling work. A single service vends the priorities back, so the priorities agree, the supply chain works as a distributed priority queue, and high-risk problems reach the team in working hours. supply chain teams call with their errors services call before scheduling work a single service vending priorities distributed priority queue the priorities agree high-risk problems in working hours One severity service for the whole supply chain Supply chain teams call with their errors, and services call before scheduling work. A single service vends the priorities back, so the priorities agree, the supply chain works as a distributed priority queue, and high-risk problems reach the team in working hours. supply chain teams call with their errors services call before scheduling work a single service vending priorities distributed priority queue the priorities agree high-risk problems in working hours
Supply chain teams call with their errors, and services call before scheduling work. A single service vends the priorities back, so the priorities agree, the supply chain works as a distributed priority queue, and high-risk problems reach the team in working hours.

I owned the automated agentic system and its monitoring and visibility. An agent did as many basic checks and redrives, retrying a step that had failed, as it could on its own, then reported to a judge that knew, from past experience, where to route the rest to the operators and specialists who had to make a human judgment. The agents also found missing file deliveries and escalated them to content partners within the agreed response time.

The failure that showed up somewhere else

The hardest failures didn't show where they started. The biggest friction was with the teams whose services failed on bad data from upstream. A title's data settled over time, and an error depended on both where a title was shipping and what content had been delivered, so the error only existed at the moment of publishing verification. It showed up in a team that hadn't caused it, and nobody could see it was the same failure again.

In parallel, I owned the supply chain's incident review process for the VP. This was important because it gave me visibility into the entire Prime Video supply chain: I could see problems and relate them to each other, and that helped shape the product and the priorities. I used it to show how many titles were failing the same way. That also showed why the recommended fix was the right call.

To show where, I built an end-to-end verification platform on a message bus that measured the latency of each leg of the supply chain. A team could see exactly which leg of a title's journey didn't meet its agreed latency. That made the case easy to put in numbers, so the team could argue for the work up its own management chain. It worked outside Prime Video too, with the teams that handled asset encoding: we could name the video formats and point to specific examples, and so prioritize the encoding formats Prime Video's business lines needed.

Each team had to onboard, which took about one to five days of its time, and getting that prioritized was hard. Where I had worked in the code before, I wrote the integration myself, in about a day. Elsewhere I turned earlier onboardings into guides and sample code that made it simple for a team to do it themselves. The monitoring then showed what was missing or not working, and I drove the fixes: faster encoding, better parallel encoding, optimized file ingestion, and parallel processing for more configurations.

Who it was for

The tooling was for the operators and specialists. Before it, they lost three quarters or more of their time to routine checks that passed, since most content did ship on time. About three quarters of what they checked went away. What was left came to them as a report of the checks that were failing: the tooling ran the checks and highlighted each failure, and they spent about 5% of their time confirming a flagged failure before acting on it. That left enough lead time to fix an issue before a launch was missed.

We didn't automate away the jobs. We improved the value of everyone by giving them more valuable work and multiplying their skills.

MeasureBeforeAfter
Highest value content live within 90 seconds of launch80%99.5%
Specialists' time on routine checks that passed75% or moreNone. About 5% went to confirming a flagged failure

Feedback that keeps up

I used that first-year launch, and others over the years, to bring teams and organizations on board with why automated monitoring was critical, and why solving each issue was not enough. The business and the customer experience move fast, so publishing requirements, quality and formats move just as fast. The content delivery industry moved slowly. We needed quick, accurate feedback on which rules, features and changes were blocking titles that hadn't been blocked before, so that behavior could keep up with the product requirements.

Want the details?

Ask the window anything this page skips.