The problem

Every architecture diagram says “highly available”. Very few teams have actually proved it. Chaos engineering replaces hope with evidence: inject a realistic failure, observe, and fix what breaks before customers find it. The hard part is doing it safely, and that is what this project solves.

What you’ll build

• A production-shaped web tier: ALB + multi-AZ Auto Scaling group with ELB health checks and target-tracking scaling
• Four AWS Fault Injection Service experiments: terminate 50% of instances, 100% CPU stress, +400 ms network latency, and a full AZ network blackout
• An automatic stop condition: if the 5xx error rate passes 20% for two minutes, every experiment halts
• A CloudWatch dashboard for error rate, healthy hosts, p95 latency and request volume

Architecture

Internet ─▶ ALB ─▶ Auto Scaling group (2–4 instances, 2 AZs, ELB health checks)
                     ▲
     FIS experiments │  terminate 50% · CPU stress · +400ms latency · AZ network blackout
                     │
CloudWatch alarm (5xx rate > 20% for 2 min) ──▶ STOPS every experiment + SNS alert
Dashboard: error rate · healthy hosts · p95 latency · requests

AWS services used

How it works

1. HypothesizeWrite down what should happen, e.g. “losing half the fleet causes <1% errors and recovers in under 3 minutes”.
2. InjectStart an FIS experiment from the CLI. FIS uses SSM documents for CPU and latency faults and native actions for termination and network disruption.
3. ObserveWatch the dashboard. Record time-to-detect, time-to-recover and peak error rate.
4. ImproveTune health-check thresholds, deregistration delay or minimum capacity, then run again until the hypothesis holds.

Key design decisions

• The stop-condition alarm is non-negotiable. It is what makes chaos safe to run during business hours.
• Experiments target instances by tag, so nothing outside the lab can be hit by accident.
• Public subnets with ALB-only security groups keep the lab cheap (no NAT gateway) without exposing instances.

Deploy it

npm install
npx cdk deploy -c alertEmail=you@example.com

Full prerequisites, parameters and clean-up steps are in the project README on the code page.

See it work

Run the AzBlackout experiment while a load loop hits the app. Watch the healthy-host graph drop and recover while the error rate stays under the stop threshold.

Skills this demonstrates