Chaos Lab
AWS FIS-style fault injection experiments for SLA validation and resilience testing
Target SLA Availability
99.99%
Recovery Time Objective
< 450ms
Chaos Test Score
100 / 100
Experiment Catalog
🟢
Baseline Normal Operation
● ACTIVERestores standard baseline operational flow. Webhooks pass through XGBoost anomaly triage with 14ms direct-pass delivery.
→ Normal operational mode (Anomaly score < 0.05)
⚡
5,000 TPS Payment Surge
RUN TESTSimulates a massive Black Friday-style payment surge. KEDA provisions Triton GPU inference nodes to handle the load.
→ Triggers GPU scale-up, high throughput mode
🔴
Worker Node Hardware Failure
RUN TESTSimulates a critical memory leak and DB lock on Node-2. Isolation Forest detects the anomaly and evicts workloads.
→ AI intercepts all payloads → Celery queue
🌙
Off-Peak Nighttime Scaling
RUN TESTSimulates low-traffic nighttime conditions (10 TPS). KEDA scales Triton to zero. CPU fallback takes over.
→ GPU shutdown → Infrastructure cost savings accelerate
⏱️
500ms Network Bottleneck
RUN TESTInjects high-latency conditions to test LSTM prediction of recovery windows and smart retry backoff behavior.
→ LSTM adjusts retry schedule in real-time
Manual Load Generator
IDLE
No traffic — drag slider to start
0 (idle)500 → GPU5,000
⏸ Simulation paused. No webhooks are being processed.