CI/CD: from pull request to production
How a change moves from a pull request to production on dansabin.com, with a live environment for every pull request, tests at every stage and all of its infrastructure defined as code in AWS CDK.
Key points
Every code pull request gets its own working copy of the site on AWS, built from the same stack definitions as production and removed when the pull request closes.
A change passes through unit tests, a real deployment and tests against that deployment before it can merge. The merge then runs the unit tests again, deploys production and checks it.
The system's AWS infrastructure is defined in AWS CDK. Application code ships only through those stacks, so code and the infrastructure it depends on always deploy together.
The pipeline is built directly on AWS primitives and GitHub Actions, with no deployment platform in between.
The design
The pipeline answers one question at each stage: is there enough evidence to move this change closer to production? Fast checks run first, and more expensive ones run only after they pass.
Three ideas shape it:
Test against real infrastructure. Templates and mocks can't show that a CloudFront distribution, a firewall rule and a Lambda function work together, so each pull request deploys them for real.
Ship what was tested. Each run builds one deployment artifact after its tests pass, and the deploy job uses that artifact rather than building again. My idea was to speed up builds and also make sure the tested bytes are the same as the bytes we deploy.
Keep every environment the same shape. Production, pull request and personal environments come from the same code. Configuration sets what differs: regions, retention and termination protection.
What it's built on
The system runs just above the bare metal of the cloud. That means AWS primitives, with the deployment logic defined in this repository. An app that calls a model API with a key can inherit its hosting, previews and deploys from a platform. Here, those are part of what was built.
| Primitive, managed by GitHub or AWS | Built on top, owned in this system |
|---|---|
| GitHub Actions runners and scheduling | The workflows: stages, ordering and what blocks what |
| GitHub rulesets | The merge policy and its one required check |
| GitHub OIDC and AWS STS | Short-lived CI access with no stored AWS keys |
| CloudFormation and the CDK libraries | Eight kinds of stack in TypeScript, with their dependencies and deploy concurrency |
| CloudFront, WAF, Route 53, ACM | Routing and gate functions, firewall rules, hostnames for every environment |
| Lambda, DynamoDB, S3, SQS, Bedrock | Application code, data design, permissions and alarms |
| Cucumber, WebdriverIO, headless Chrome | A Gherkin test harness with its own steps, shared test data store and Features |
AWS still does the hard physical work: applying changes in order, validating certificates and propagating distributions to the edge. The system decides what exists, when it changes and what must pass first.
Pull request stage
The first job reads the changed paths. A pull request that touches only specs or documentation gets a spec lint and deploys nothing. Anything else runs three test suites side by side: infrastructure, API and web, and harness and scripts. The infrastructure suite also builds the application and synthesizes the stacks. That step needs no AWS credentials, because the one lookup it depends on is committed to the repository.
Once all three suites pass, the deploy job signs in through OIDC, confirms that public DNS still points at the system's hosted zone, and deploys the artifact. The pull request gets a comment with its environment's URLs, and test reports are saved with the run.
Tests then run against that live environment, in order. First, 18 Gherkin Scenarios use HTTP calls and headless Chrome. Next, one firewall Scenario runs on its own, because it deliberately trips the rate limit. Last, a routing check reads the live routing function and test-runs it in CloudFront's own runtime.
A single required check summarizes every stage, so documentation and code changes can share one merge rule. Main accepts only squash merges through a pull request with that check green, and it blocks force pushes.
Production stage
A merge to main that changes more than documentation starts the production workflow. Runs queue rather than overlap. The three test suites run again on the merged commit, because a branch doesn't have to be up to date with main before it merges. The run then builds production's own artifact and deploys it, up to four stacks at once, followed by 13 read-only smoke Scenarios and the routing check. The multi-region page covers what that check protects.
CloudFormation reverts a stack whose own update fails. Rolling back a deploy that succeeded is semi-automatic: alarms go off, I decide, and reverting the change deploys the previous version of production by itself. Full automation could be added, and the reason it hasn't been is the product. A bake time plus rollback interrupts a massive outage by checking for increased error rates after a deploy, and with no external customers, none of the site's traffic would cause that at the moment. API customers would change that.
Stacks as code
Every environment is the same set of stacks, named by a suffix: prod, a personal one per developer, and one per pull request. The teardowns are important since personal and ephemeral stacks need to be able to be worked on and iterated and they are regularly deleted. Small security choices live in the same code, such as answering 404 rather than 403, so nobody can discover what exists by walking paths.
| Stack | Holds | Scope | Protection |
|---|---|---|---|
| DNS | Hosted zone, shared dev certificate, CAPTCHA key | Account | Termination protection always on; zone and key retained |
| Operations (two kinds) | Budget, alerts, kill switch, email sending | Account, and one per extra region | Termination protection always on |
| CI access | GitHub OIDC provider and deploy role | Account | Termination protection on |
| Hosting | Site bucket in the primary region, distribution, DNS records | Environment | From configuration |
| API edge | Distribution, firewall, routing, failover | Environment | From configuration |
| Region | Functions, table, buckets, Knowledge Base, alarms | Environment and region | From configuration |
| Region links | Transcript replication between regions, and the second site bucket | Environment and region | From configuration |
Production turns on retention and termination protection. Pull request and personal environments turn them off, so they delete cleanly. CI deploys only environment stacks, and environment stacks hold no shared resources, so tearing one down leaves DNS and the other shared stacks in place.
A pull request environment is one region of the same stacks. It imports the shared dev certificate instead of requesting its own, and the API and hosting stacks deploy side by side. Those two choices cut a new environment's deploy step from a median of 16.1 minutes to 8.5. It now sits near 10 minutes, and an update to an existing environment usually takes one to three. The median full pull request run is about 18 minutes, and the median of the last 20 production runs is about 12.
Test layers
| Layer | What it proves | Size | Runs |
|---|---|---|---|
| Infrastructure | Synthesized templates, wiring, naming, edge function behavior | About 880 tests | Before push, pull request, production |
| API | Lambda logic | About 1,800 tests | Before push, pull request, production |
| Web | User interface behavior | About 2,900 tests | Before push, pull request, production |
| Harness and scripts | The test runner and operational scripts | About 930 tests | Before push, pull request, production |
| Live environment | The deployed system's behavior | 19 Scenarios | Pull request |
| Smoke and routing | Production answers correctly after a deploy | 13 Scenarios, routing check | Production |
| Mock API | Interface behavior against a stand-in API | 23 Scenarios | Pull request, not blocking a merge |
The infrastructure, API and web suites enforce line coverage floors of 80, 85 and 60 percent, and each measures above 94 on the latest production run. A local pre-push hook runs the unit suites for fast feedback, and CI repeats them, so the pipeline doesn't depend on it. Agent evaluations cost real model tokens and run separately, as described on the agent page. The testing page covers why the tests look like this.
Design decisions and trade-offs
| Decision | Benefit | Trade-off |
|---|---|---|
| A full environment per pull request | Real tests of deployed behavior, and a live copy to review | Each new environment adds about 10 minutes and runs real AWS resources |
| One region per pull request environment | Faster, simpler environments | Two-region behavior isn't exercised before merge; failover drills are run separately |
| Parallel test suites ahead of the deploy | No environment is updated with failing tests | The slowest suite sets the deploy's start |
| Tests again after merge | Catches anything that changes when branches combine | Adds about 6 minutes to each production run |
| Both regions in one automated deploy | Simple, with one version live everywhere | No canary or bake time, so a bad change reaches both regions |
| Semi-automatic rollback, with no bake time | There are no external customers, and it saves cost | A bad deploy stays live until someone reverts it |
| Read-only smoke tests | Safe to run against production on every deploy | They don't run a chat turn or solve the human check |
| Function URLs, not API Gateway | API Gateway's integration timeout defaults to 29 seconds, and raising it takes a quota request; longer agent answers could exceed it even while sending data | The system builds its own signing and routing at the edge |
| Personal and pull request environments in the prod account, for now | One account to run, for now | They aren't isolated from prod, and a separate dev account is the better home |
| Shared stacks deployed by an operator | Routine deploys don't touch DNS or budgets | A merged change to them isn't live until someone deploys it |
| Guards written in code and tests | Reviewable and versioned with the system | An operator deploy from an out-of-date checkout runs that checkout's older code and tests |
Want the details?
Ask the window anything this page skips.