Multi-region design
How dansabin.com keeps answering when a region fails, with an active-active API in two AWS regions, chosen per request at the CloudFront edge, pages served worldwide through a global CDN, and data that follows the visitor.
Key points
The API is active-active in us-east-1 and us-west-2. Each request goes to the nearest healthy region, chosen at the CloudFront edge.
The app's pages are delivered worldwide through CloudFront, so the page a visitor loads doesn't depend on which API region answers.
Data follows the visitor: a DynamoDB global table, S3 transcript buckets that replicate both ways, and replicated signing keys.
When a region fails its health check, new requests move to the other region with no human involved, in about two and a half to four minutes in rehearsals on non-production copies.
The design
dansabin.com has two delivery paths with different shapes. Pages are static files, so they're served by a global CDN. The API is live compute with state, so it runs as two complete copies, one per region, and both take traffic all the time.
Pages. CloudFront serves the app from its edge locations, under its firewall and with its own certificate. The pages live in two S3 buckets behind that CDN, one in us-east-1 and one in us-west-2. Edges serve pages they hold for up to an hour and otherwise fetch them from an origin group, which moves to the other bucket per request when the first answers 500, 502, 503 or 504.
API. One hostname and one CloudFront distribution sit in front of both regions. There's no primary that sits idle and no standby to promote: a viewer near Virginia is served from us-east-1, and a viewer near Oregon from us-west-2. Each region has its own chat function, API function, cleanup function, Knowledge Base and model profile.
Data. A visitor's identity, conversations and allowances are readable in either region, so a visitor moved between regions keeps their place. A token issued in one region is accepted in the other.
Why two regions on a personal site
This site is a reference architecture for cloud infrastructure, so failover was part of the plan from the first days: regions are inputs to the configuration, and the system is built as if it were deploying regional cells.
Choosing a region per request
A CloudFront Function runs on API requests before any origin is contacted. It computes the great-circle distance from the viewer's latitude and longitude, which CloudFront supplies, to each deployed region. It then walks the regions nearest first and picks the first whose health flag is up.
The function selects that region's origin and rewrites the path, so a streaming POST needs no redirect. A request with no usable location goes to the primary region first. If every flag is down, the request goes to the nearest region anyway, so an outage appears as an outage.
The origins are Lambda Function URLs that require signed requests, and CloudFront signs them through origin access control. The function uses about 10% of its compute budget and is close to CloudFront's 10 KB size limit, so it stays deliberately small.
Detecting failure and moving traffic
Routing only reads a flag, so detection runs separately:
| Stage | Behavior | Time |
|---|---|---|
| Health checks | Every 30 seconds, two failures in a row mark the region unhealthy | About 1 minute |
| Alarm | Evaluates one-minute periods and fires on one bad minute | About 1 minute |
| Failover Lambda | Reads the alarm's current state, so a late retry can't write an old answer | Seconds |
| KeyValueStore | CloudFront copies the flag to its edge locations | Up to about 40 seconds |
Rehearsals on non-production copies of the same stacks moved traffic in 2 minutes 20 seconds to 3 minutes 56 seconds, depending on how the region was broken. Production hasn't failed over for real yet. Traffic returns only after five healthy minutes in a row, about eight minutes after the fault clears, which keeps a region from flapping.
The health endpoint checks the region's table, storage, signing keys, kill switch and Knowledge Base, along with the alarms that mean a visitor can't be served. It never calls the model. Checks reach it through CloudFront on a path that skips the routing function, so a routing bug can't mark a healthy region down.
During the switch, visitors near the failing region see errors, and the page offers a retry. A reply in progress ends early and is marked as cut short, and the retry sends the same message again. The message is stored before the model runs and a reply that fails partway is kept, so the question survives the switch. The status bar names the region that answered.
Keeping data in both regions
| Data | Store | How it reaches the other region | Timing |
|---|---|---|---|
| Identities, conversations, allowances, visits | DynamoDB global table | Managed replication; last writer wins | 0.7 seconds, one item measured |
| Transcripts and agent snapshots | One S3 bucket per region | S3 replication rule in each direction | Within 20 seconds in rehearsal |
| Signing keys | Secrets Manager | Replica; rotation runs once, in us-east-1 | Seconds, covered by a key-settling delay |
| Kill switch | Parameter Store in each region | Written to every region; read locally | Read cache of 60 seconds |
| Knowledge | Bucket, Knowledge Base and vector index per region | Each region deploys from the repository | At deploy |
Deletion needs its own handling, because S3 replication doesn't copy version deletes. A purge therefore lists and deletes every version of a visitor's transcripts in every region's bucket. If one region fails, the others are still purged and the job retries.
Checking it stays correct
After every deploy, CI test-runs the live routing function in CloudFront's own runtime with each region's location. A wrong route or a down region fails the build, with no rollback. Prod is checked this way on every deploy; the throttling rehearsals run on demand. See the CI/CD page for how that pipeline works, and the agent page for what a turn does once it reaches a region.
AWS primitives, and what's built on them
The system runs just above the bare metal of the cloud: AWS primitives, with the logic on top defined in this repository. An app that calls a model API with a key depends on that API's availability. Here, routing, failure detection, data placement and deletion are part of the system, built directly on AWS primitives and defined entirely in CDK.
| AWS provides | Built on top |
|---|---|
| CloudFront, its Functions runtime and viewer location | Distance-based region selection and origin choice on every API request |
| CloudFront KeyValueStore | Health flags, written with version-checked retries |
| Route 53 health checks and CloudWatch alarms | The health endpoint, the choice of which alarms count, and timing for failure and recovery |
| Lambda and Function URLs | Streaming chat and API handlers, deployed per region and signed at the edge |
| DynamoDB global tables | Key design, conditional writes for allowances and turn claims, the accepted conflict behavior |
| S3 and its replication | Transcript layout, two-way rules scoped to conversations, multi-region purge, backfill |
| Secrets Manager replication | The signing key ring, its cache and its rotation |
| Amazon Bedrock | Per-region model profiles, and the agent on top of them |
Design decisions and trade-offs
| Chose | Over, and why | Trade-off |
|---|---|---|
| Route inside one distribution using a flag | DNS failover between distributions, because CloudFront ties a hostname to one distribution, and over origin groups, because they don't fail over POST, which every chat turn uses. | A function runs on each API request and must stay small. |
| One deploy writes both page buckets | S3 replication, to keep the moving parts down. | The failover hasn't been tested against S3 itself answering with errors. Pull request environments keep one bucket. |
| Nearest region by distance | Route 53 latency records, which would replace the routing function, over country, which can't separate two US regions, and over time zone, which is coarser than distance. | Distance stands in for measured latency. |
| Standard health checks and one-minute alarms | Faster checks, at higher cost and with the same one-minute alarm floor. | Going by the rehearsals, visitors near a failed region see about two and a half to four minutes of errors. |
| Eventual consistency between regions | Sending every allowance write to one region, estimated at about 70 ms per Oregon turn plus a fallback rule. | In the seconds around a switch, a visitor can briefly over-spend or start a second turn; site-wide limits and the kill switch still apply. |
| Asynchronous transcript replication | Region-aware reads at request time. | A turn stored just before a region fails can be missing from the other region until the copy completes. |
| Replicated signing keys | Reading one region's secret from the other. | A small monthly cost per replica, and tokens keep verifying if a region is lost. |
| One region for the failover automation | A copy of the detection chain in each region. | Route 53 publishes its health metric only in us-east-1, so the alarm and failover Lambda run there. A us-east-1 failure that also stops them could leave its flag up and keep nearby visitors on it; visitors nearest us-west-2 are already routed there. Rehearsals haven't taken out us-east-1's control path. |
The second region costs an estimated $9.00 to $10.25 a month. Alarms are the largest item, since each region carries its own full set.
Want the details?
Ask the window anything this page skips.