Multi-region design

How dansabin.com keeps answering when a region fails, with an active-active API in two AWS regions, chosen per request at the CloudFront edge, pages served worldwide through a global CDN, and data that follows the visitor.

case study · 2026
~/dansabin.com — chat Ask about this project

Key points

API

The API is active-active in us-east-1 and us-west-2. Each request goes to the nearest healthy region, chosen at the CloudFront edge.

Pages

The app's pages are delivered worldwide through CloudFront, so the page a visitor loads doesn't depend on which API region answers.

Data

Data follows the visitor: a DynamoDB global table, S3 transcript buckets that replicate both ways, and replicated signing keys.

Failover

When a region fails its health check, new requests move to the other region with no human involved, in about two and a half to four minutes in rehearsals on non-production copies.

The design

dansabin.com has two delivery paths with different shapes. Pages are static files, so they're served by a global CDN. The API is live compute with state, so it runs as two complete copies, one per region, and both take traffic all the time.

Two active regions behind the CloudFront edge The map draws us-west-2 and us-east-1 as mirrors, with the CloudFront edge above them. Visitors enter at the edge. The routing function reads its flags from the KeyValueStore and sends each API request to one region, and the site distribution serves pages from S3 buckets in both regions. Each region holds its own functions, model profile, Knowledge Base, signing keys, table replica and transcript bucket. The lines between the regions show what crosses: the global table and S3 replication in both directions, and the signing keys from us-east-1 to us-west-2. visitors nearer Oregon visitors nearer Virginia CLOUDFRONT EDGE · GLOBAL API distribution firewall, certificate Routing function nearest healthy region KeyValueStore up or down flags Site distribution pages, worldwide US-WEST-2 · OREGON · ACTIVE US-EAST-1 · VIRGINIA · ACTIVE Chat and API functions streaming reply, Strands agent Chat and API functions streaming reply, Strands agent Bedrock and Knowledge Base own model profile and vector index Bedrock and Knowledge Base own model profile and vector index Signing keys replica Signing keys rotated here DynamoDB replica of the global table DynamoDB replica of the global table S3 transcripts this region's own bucket S3 transcripts this region's own bucket S3 site bucket the second copy of the pages S3 site bucket the pages, both copies in one deploy fallback key replica global table S3 replication Two active regions behind the CloudFront edge The map draws us-west-2 and us-east-1 as mirrors, with the CloudFront edge above them. Visitors enter at the edge. The routing function reads its flags from the KeyValueStore and sends each API request to one region, and the site distribution serves pages from S3 buckets in both regions. Each region holds its own functions, model profile, Knowledge Base, signing keys, table replica and transcript bucket. The lines between the regions show what crosses: the global table and S3 replication in both directions, and the signing keys from us-east-1 to us-west-2. visitors CLOUDFRONT EDGE · GLOBAL API distribution firewall, certificate Routing function nearest healthy region KeyValueStore up or down flags Site distribution pages, worldwide US-WEST-2 · OREGON · ACTIVE Chat and API functions streaming reply, Strands agent Bedrock and Knowledge Base own model profile and vector index Signing keys replica DynamoDB replica of the global table S3 transcripts this region's own bucket keys global table S3 replication US-EAST-1 · VIRGINIA · ACTIVE Chat and API functions streaming reply, Strands agent Bedrock and Knowledge Base own model profile and vector index Signing keys rotated here DynamoDB replica of the global table S3 transcripts this region's own bucket S3 site bucket the pages, a copy in each region
The map draws us-west-2 and us-east-1 as mirrors, with the CloudFront edge above them. Visitors enter at the edge. The routing function reads its flags from the KeyValueStore and sends each API request to one region, and the site distribution serves pages from S3 buckets in both regions. Each region holds its own functions, model profile, Knowledge Base, signing keys, table replica and transcript bucket. The lines between the regions show what crosses: the global table and S3 replication in both directions, and the signing keys from us-east-1 to us-west-2.

Pages. CloudFront serves the app from its edge locations, under its firewall and with its own certificate. The pages live in two S3 buckets behind that CDN, one in us-east-1 and one in us-west-2. Edges serve pages they hold for up to an hour and otherwise fetch them from an origin group, which moves to the other bucket per request when the first answers 500, 502, 503 or 504.

API. One hostname and one CloudFront distribution sit in front of both regions. There's no primary that sits idle and no standby to promote: a viewer near Virginia is served from us-east-1, and a viewer near Oregon from us-west-2. Each region has its own chat function, API function, cleanup function, Knowledge Base and model profile.

Data. A visitor's identity, conversations and allowances are readable in either region, so a visitor moved between regions keeps their place. A token issued in one region is accepted in the other.

Why two regions on a personal site

This site is a reference architecture for cloud infrastructure, so failover was part of the plan from the first days: regions are inputs to the configuration, and the system is built as if it were deploying regional cells.

Choosing a region per request

A CloudFront Function runs on API requests before any origin is contacted. It computes the great-circle distance from the viewer's latitude and longitude, which CloudFront supplies, to each deployed region. It then walks the regions nearest first and picks the first whose health flag is up.

The function selects that region's origin and rewrites the path, so a streaming POST needs no redirect. A request with no usable location goes to the primary region first. If every flag is down, the request goes to the nearest region anyway, so an outage appears as an outage.

The origins are Lambda Function URLs that require signed requests, and CloudFront signs them through origin access control. The function uses about 10% of its compute budget and is close to CloudFront's 10 KB size limit, so it stays deliberately small.

Detecting failure and moving traffic

Routing only reads a flag, so detection runs separately:

A failover as a sequence The sequence follows one failure through the chain. Route 53's checks fail against the region, and the CloudWatch alarm fires. The failover Lambda writes the flag to the KeyValueStore, and CloudFront copies it to the edge. The next request reaches the routing function, which sends it to the nearest region whose flag is up, and the reply streams back. When the failed region recovers, the same chain runs again and puts the flag back up. visitor browser routing function KeyValueStore flags failover Lambda CloudWatch alarm Route 53 checks failed region healthy region A REGION FAILS two failed checks unhealthy fires flag: down copied to the edge THE NEXT REQUEST request nearest region whose flag is up the reply streams THE REGION RECOVERS checks pass healthy clears after five healthy minutes flag: up A failover as a sequence The sequence follows one failure through the chain. Route 53's checks fail against the region, and the CloudWatch alarm fires. The failover Lambda writes the flag to the KeyValueStore, and CloudFront copies it to the edge. The next request reaches the routing function, which sends it to the nearest region whose flag is up, and the reply streams back. When the failed region recovers, the same chain runs again and puts the flag back up. Route 53 health checks two failures, 30 seconds apart CloudWatch alarm fires on one bad minute Failover Lambda reads the alarm, writes the flag KeyValueStore CloudFront copies it to the edge Routing function skips the region marked down Healthy region serves the next request THE REGION RECOVERS Five healthy minutes later the alarm clears, the flag goes up
The sequence follows one failure through the chain. Route 53's checks fail against the region, and the CloudWatch alarm fires. The failover Lambda writes the flag to the KeyValueStore, and CloudFront copies it to the edge. The next request reaches the routing function, which sends it to the nearest region whose flag is up, and the reply streams back. When the failed region recovers, the same chain runs again and puts the flag back up.
StageBehaviorTime
Health checksEvery 30 seconds, two failures in a row mark the region unhealthyAbout 1 minute
AlarmEvaluates one-minute periods and fires on one bad minuteAbout 1 minute
Failover LambdaReads the alarm's current state, so a late retry can't write an old answerSeconds
KeyValueStoreCloudFront copies the flag to its edge locationsUp to about 40 seconds

Rehearsals on non-production copies of the same stacks moved traffic in 2 minutes 20 seconds to 3 minutes 56 seconds, depending on how the region was broken. Production hasn't failed over for real yet. Traffic returns only after five healthy minutes in a row, about eight minutes after the fault clears, which keeps a region from flapping.

The health endpoint checks the region's table, storage, signing keys, kill switch and Knowledge Base, along with the alarms that mean a visitor can't be served. It never calls the model. Checks reach it through CloudFront on a path that skips the routing function, so a routing bug can't mark a healthy region down.

During the switch, visitors near the failing region see errors, and the page offers a retry. A reply in progress ends early and is marked as cut short, and the retry sends the same message again. The message is stored before the model runs and a reply that fails partway is kept, so the question survives the switch. The status bar names the region that answered.

Keeping data in both regions

DataStoreHow it reaches the other regionTiming
Identities, conversations, allowances, visitsDynamoDB global tableManaged replication; last writer wins0.7 seconds, one item measured
Transcripts and agent snapshotsOne S3 bucket per regionS3 replication rule in each directionWithin 20 seconds in rehearsal
Signing keysSecrets ManagerReplica; rotation runs once, in us-east-1Seconds, covered by a key-settling delay
Kill switchParameter Store in each regionWritten to every region; read locallyRead cache of 60 seconds
KnowledgeBucket, Knowledge Base and vector index per regionEach region deploys from the repositoryAt deploy

Deletion needs its own handling, because S3 replication doesn't copy version deletes. A purge therefore lists and deletes every version of a visitor's transcripts in every region's bucket. If one region fails, the others are still purged and the job retries.

Checking it stays correct

After every deploy, CI test-runs the live routing function in CloudFront's own runtime with each region's location. A wrong route or a down region fails the build, with no rollback. Prod is checked this way on every deploy; the throttling rehearsals run on demand. See the CI/CD page for how that pipeline works, and the agent page for what a turn does once it reaches a region.

AWS primitives, and what's built on them

The system runs just above the bare metal of the cloud: AWS primitives, with the logic on top defined in this repository. An app that calls a model API with a key depends on that API's availability. Here, routing, failure detection, data placement and deletion are part of the system, built directly on AWS primitives and defined entirely in CDK.

AWS providesBuilt on top
CloudFront, its Functions runtime and viewer locationDistance-based region selection and origin choice on every API request
CloudFront KeyValueStoreHealth flags, written with version-checked retries
Route 53 health checks and CloudWatch alarmsThe health endpoint, the choice of which alarms count, and timing for failure and recovery
Lambda and Function URLsStreaming chat and API handlers, deployed per region and signed at the edge
DynamoDB global tablesKey design, conditional writes for allowances and turn claims, the accepted conflict behavior
S3 and its replicationTranscript layout, two-way rules scoped to conversations, multi-region purge, backfill
Secrets Manager replicationThe signing key ring, its cache and its rotation
Amazon BedrockPer-region model profiles, and the agent on top of them

Design decisions and trade-offs

ChoseOver, and whyTrade-off
Route inside one distribution using a flagDNS failover between distributions, because CloudFront ties a hostname to one distribution, and over origin groups, because they don't fail over POST, which every chat turn uses.A function runs on each API request and must stay small.
One deploy writes both page bucketsS3 replication, to keep the moving parts down.The failover hasn't been tested against S3 itself answering with errors. Pull request environments keep one bucket.
Nearest region by distanceRoute 53 latency records, which would replace the routing function, over country, which can't separate two US regions, and over time zone, which is coarser than distance.Distance stands in for measured latency.
Standard health checks and one-minute alarmsFaster checks, at higher cost and with the same one-minute alarm floor.Going by the rehearsals, visitors near a failed region see about two and a half to four minutes of errors.
Eventual consistency between regionsSending every allowance write to one region, estimated at about 70 ms per Oregon turn plus a fallback rule.In the seconds around a switch, a visitor can briefly over-spend or start a second turn; site-wide limits and the kill switch still apply.
Asynchronous transcript replicationRegion-aware reads at request time.A turn stored just before a region fails can be missing from the other region until the copy completes.
Replicated signing keysReading one region's secret from the other.A small monthly cost per replica, and tokens keep verifying if a region is lost.
One region for the failover automationA copy of the detection chain in each region.Route 53 publishes its health metric only in us-east-1, so the alarm and failover Lambda run there. A us-east-1 failure that also stops them could leave its flag up and keep nearby visitors on it; visitors nearest us-west-2 are already routed there. Rehearsals haven't taken out us-east-1's control path.

The second region costs an estimated $9.00 to $10.25 a month. Alarms are the largest item, since each region carries its own full set.

Want the details?

Ask the window anything this page skips.