CI/CD: from pull request to production

How a change moves from a pull request to production on dansabin.com, with a live environment for every pull request, tests at every stage and all of its infrastructure defined as code in AWS CDK.

case study · 2026
~/dansabin.com — chat Ask about this project

Key points

Environments, per pull request

Every code pull request gets its own working copy of the site on AWS, built from the same stack definitions as production and removed when the pull request closes.

Path to production, gated stages

A change passes through unit tests, a real deployment and tests against that deployment before it can merge. The merge then runs the unit tests again, deploys production and checks it.

Stacks as code, AWS CDK

The system's AWS infrastructure is defined in AWS CDK. Application code ships only through those stacks, so code and the infrastructure it depends on always deploy together.

Foundation, AWS primitives

The pipeline is built directly on AWS primitives and GitHub Actions, with no deployment platform in between.

The design

The pipeline answers one question at each stage: is there enough evidence to move this change closer to production? Fast checks run first, and more expensive ones run only after they pass.

The CI/CD pipeline, from pull request to production The pipeline has two lanes. On a pull request, changed files are classified first. Specs or documentation go to spec lint. Code goes to three parallel test suites, then a deployed pull request environment, then live tests. Both paths meet at the required check. A squash merge to main runs the test suites again, deploys production to both regions, then runs smoke tests and the routing check. PULL REQUEST Classify changed files code PARALLEL Infrastructure API and web Harness and scripts Deploy live environment Live tests Scenarios, routing Required check every stage green specs or docs Spec lint squash merge MAIN Main branch merged commit Test suites again on the merged commit Deploy production both regions Smoke and routing 13 read-only Scenarios rollback: a manual decision The CI/CD pipeline, from pull request to production The pipeline has two lanes. On a pull request, changed files are classified first. Specs or documentation go to spec lint. Code goes to three parallel test suites, then a deployed pull request environment, then live tests. Both paths meet at the required check. A squash merge to main runs the test suites again, deploys production to both regions, then runs smoke tests and the routing check. PULL REQUEST Classify changed files code Infrastructure API and web Harness and scripts Deploy live environment Live tests Scenarios, routing Spec lint no deploy Required check every stage green squash merge MAIN Main branch merged commit Test suites again on the merged commit Deploy production both regions Smoke and routing 13 read-only Scenarios rollback: a manual decision
The pipeline has two lanes. On a pull request, changed files are classified first. Specs or documentation go to spec lint. Code goes to three parallel test suites, then a deployed pull request environment, then live tests. Both paths meet at the required check. A squash merge to main runs the test suites again, deploys production to both regions, then runs smoke tests and the routing check.

Three ideas shape it:

Test against real infrastructure. Templates and mocks can't show that a CloudFront distribution, a firewall rule and a Lambda function work together, so each pull request deploys them for real.

Ship what was tested. Each run builds one deployment artifact after its tests pass, and the deploy job uses that artifact rather than building again. My idea was to speed up builds and also make sure the tested bytes are the same as the bytes we deploy.

Keep every environment the same shape. Production, pull request and personal environments come from the same code. Configuration sets what differs: regions, retention and termination protection.

What it's built on

The system runs just above the bare metal of the cloud. That means AWS primitives, with the deployment logic defined in this repository. An app that calls a model API with a key can inherit its hosting, previews and deploys from a platform. Here, those are part of what was built.

Primitive, managed by GitHub or AWSBuilt on top, owned in this system
GitHub Actions runners and schedulingThe workflows: stages, ordering and what blocks what
GitHub rulesetsThe merge policy and its one required check
GitHub OIDC and AWS STSShort-lived CI access with no stored AWS keys
CloudFormation and the CDK librariesEight kinds of stack in TypeScript, with their dependencies and deploy concurrency
CloudFront, WAF, Route 53, ACMRouting and gate functions, firewall rules, hostnames for every environment
Lambda, DynamoDB, S3, SQS, BedrockApplication code, data design, permissions and alarms
Cucumber, WebdriverIO, headless ChromeA Gherkin test harness with its own steps, shared test data store and Features

AWS still does the hard physical work: applying changes in order, validating certificates and propagating distributions to the edge. The system decides what exists, when it changes and what must pass first.

Pull request stage

The first job reads the changed paths. A pull request that touches only specs or documentation gets a spec lint and deploys nothing. Anything else runs three test suites side by side: infrastructure, API and web, and harness and scripts. The infrastructure suite also builds the application and synthesizes the stacks. That step needs no AWS credentials, because the one lookup it depends on is committed to the repository.

Once all three suites pass, the deploy job signs in through OIDC, confirms that public DNS still points at the system's hosted zone, and deploys the artifact. The pull request gets a comment with its environment's URLs, and test reports are saved with the run.

Tests then run against that live environment, in order. First, 18 Gherkin Scenarios use HTTP calls and headless Chrome. Next, one firewall Scenario runs on its own, because it deliberately trips the rate limit. Last, a routing check reads the live routing function and test-runs it in CloudFront's own runtime.

A single required check summarizes every stage, so documentation and code changes can share one merge rule. Main accepts only squash merges through a pull request with that check green, and it blocks force pushes.

Production stage

A merge to main that changes more than documentation starts the production workflow. Runs queue rather than overlap. The three test suites run again on the merged commit, because a branch doesn't have to be up to date with main before it merges. The run then builds production's own artifact and deploys it, up to four stacks at once, followed by 13 read-only smoke Scenarios and the routing check. The multi-region page covers what that check protects.

CloudFormation reverts a stack whose own update fails. Rolling back a deploy that succeeded is semi-automatic: alarms go off, I decide, and reverting the change deploys the previous version of production by itself. Full automation could be added, and the reason it hasn't been is the product. A bake time plus rollback interrupts a massive outage by checking for increased error rates after a deploy, and with no external customers, none of the site's traffic would cause that at the moment. API customers would change that.

Stacks as code

Every environment is the same set of stacks, named by a suffix: prod, a personal one per developer, and one per pull request. The teardowns are important since personal and ephemeral stacks need to be able to be worked on and iterated and they are regularly deleted. Small security choices live in the same code, such as answering 404 rather than 403, so nobody can discover what exists by walking paths.

Account-wide stacks and environment stacks Stacks fall into two groups. Account-wide stacks hold DNS, operations and CI access, and are deployed by an operator. Environment stacks are deployed by CI or an operator and read values from the account-wide stacks. In each environment, the region stacks deploy first, followed by hosting, the API edge and region links. ACCOUNT-WIDE deployed by an operator DNS zone, dev certificate, CAPTCHA key Operations budget, alerts, kill switch CI access GitHub OIDC, deploy role read by PER ENVIRONMENT deployed by CI or an operator Region stacks functions, data, alarms Hosting site bucket, distribution API edge firewall, routing, failover Region links transcript replication after Account-wide stacks and environment stacks Stacks fall into two groups. Account-wide stacks hold DNS, operations and CI access, and are deployed by an operator. Environment stacks are deployed by CI or an operator and read values from the account-wide stacks. In each environment, the region stacks deploy first, followed by hosting, the API edge and region links. ACCOUNT-WIDE deployed by an operator DNS zone, dev certificate, CAPTCHA key Operations budget, alerts, kill switch CI access GitHub OIDC, deploy role read by PER ENVIRONMENT deployed by CI or an operator Region stacks functions, data, alarms then Hosting site bucket, distribution API edge firewall, routing, failover Region links transcript replication
Stacks fall into two groups. Account-wide stacks hold DNS, operations and CI access, and are deployed by an operator. Environment stacks are deployed by CI or an operator and read values from the account-wide stacks. In each environment, the region stacks deploy first, followed by hosting, the API edge and region links.
StackHoldsScopeProtection
DNSHosted zone, shared dev certificate, CAPTCHA keyAccountTermination protection always on; zone and key retained
Operations (two kinds)Budget, alerts, kill switch, email sendingAccount, and one per extra regionTermination protection always on
CI accessGitHub OIDC provider and deploy roleAccountTermination protection on
HostingSite bucket in the primary region, distribution, DNS recordsEnvironmentFrom configuration
API edgeDistribution, firewall, routing, failoverEnvironmentFrom configuration
RegionFunctions, table, buckets, Knowledge Base, alarmsEnvironment and regionFrom configuration
Region linksTranscript replication between regions, and the second site bucketEnvironment and regionFrom configuration

Production turns on retention and termination protection. Pull request and personal environments turn them off, so they delete cleanly. CI deploys only environment stacks, and environment stacks hold no shared resources, so tearing one down leaves DNS and the other shared stacks in place.

A pull request environment is one region of the same stacks. It imports the shared dev certificate instead of requesting its own, and the API and hosting stacks deploy side by side. Those two choices cut a new environment's deploy step from a median of 16.1 minutes to 8.5. It now sits near 10 minutes, and an update to an existing environment usually takes one to three. The median full pull request run is about 18 minutes, and the median of the last 20 production runs is about 12.

Test layers

LayerWhat it provesSizeRuns
InfrastructureSynthesized templates, wiring, naming, edge function behaviorAbout 880 testsBefore push, pull request, production
APILambda logicAbout 1,800 testsBefore push, pull request, production
WebUser interface behaviorAbout 2,900 testsBefore push, pull request, production
Harness and scriptsThe test runner and operational scriptsAbout 930 testsBefore push, pull request, production
Live environmentThe deployed system's behavior19 ScenariosPull request
Smoke and routingProduction answers correctly after a deploy13 Scenarios, routing checkProduction
Mock APIInterface behavior against a stand-in API23 ScenariosPull request, not blocking a merge

The infrastructure, API and web suites enforce line coverage floors of 80, 85 and 60 percent, and each measures above 94 on the latest production run. A local pre-push hook runs the unit suites for fast feedback, and CI repeats them, so the pipeline doesn't depend on it. Agent evaluations cost real model tokens and run separately, as described on the agent page. The testing page covers why the tests look like this.

Design decisions and trade-offs

DecisionBenefitTrade-off
A full environment per pull requestReal tests of deployed behavior, and a live copy to reviewEach new environment adds about 10 minutes and runs real AWS resources
One region per pull request environmentFaster, simpler environmentsTwo-region behavior isn't exercised before merge; failover drills are run separately
Parallel test suites ahead of the deployNo environment is updated with failing testsThe slowest suite sets the deploy's start
Tests again after mergeCatches anything that changes when branches combineAdds about 6 minutes to each production run
Both regions in one automated deploySimple, with one version live everywhereNo canary or bake time, so a bad change reaches both regions
Semi-automatic rollback, with no bake timeThere are no external customers, and it saves costA bad deploy stays live until someone reverts it
Read-only smoke testsSafe to run against production on every deployThey don't run a chat turn or solve the human check
Function URLs, not API GatewayAPI Gateway's integration timeout defaults to 29 seconds, and raising it takes a quota request; longer agent answers could exceed it even while sending dataThe system builds its own signing and routing at the edge
Personal and pull request environments in the prod account, for nowOne account to run, for nowThey aren't isolated from prod, and a separate dev account is the better home
Shared stacks deployed by an operatorRoutine deploys don't touch DNS or budgetsA merged change to them isn't live until someone deploys it
Guards written in code and testsReviewable and versioned with the systemAn operator deploy from an out-of-date checkout runs that checkout's older code and tests

Want the details?

Ask the window anything this page skips.