Chaos Engineering & Resilience Engineering Notes
Interview Summary:
Chaos Engineering helps uncover hidden production failures before customers experience them.
The goal is to improve system reliability, resiliency, observability, and recovery mechanisms.
1. Why Chaos Engineering?
- Identify hidden production risks
- Validate failover mechanisms
- Test resiliency under failure conditions
- Improve incident response
- Increase confidence in deployments
- Prevent cascading failures
2. Failure Scenarios Tested
Network-Level Failures
- Blackhole Testing – Drop all traffic to simulate service outage.
- Latency Injection – Introduce network delays.
- Packet Loss – Simulate unstable networks.
- DNS Failure – Test service discovery failures.
Infrastructure-Level Failures
- CPU Stress Testing
- Memory Pressure Testing
- Thread Pool Exhaustion
- Disk Utilization Testing
Application-Level Failures
- Database Connection Pool Exhaustion
- Max Concurrent Requests
- Queue Overflow
- Rate Limiting Thresholds
- Dependency Service Failure
3. Observed Problems
- Retry Storms
- Cascading Failures
- Thread Starvation
- Connection Pool Saturation
- Slow Dependency Responses
- Auto-scaling Delays
- Message Duplication
4. Resilience Patterns
| Pattern |
Purpose |
| Circuit Breaker |
Stops repeated calls to failing services |
| Retry |
Handles transient failures |
| Exponential Backoff |
Controls retry frequency |
| Timeout |
Prevents thread blocking |
| Bulkhead |
Isolates resources |
| Rate Limiter |
Controls traffic volume |
| Fallback |
Provides alternate response |
| Cache |
Reduces repeated calls |
| TimeLimiter |
Limits execution time |
| Graceful Degradation |
Keeps critical services operational |
5. Bulkhead Types
Semaphore Bulkhead
- Limits concurrent requests.
- Rejects excess traffic immediately.
- Best for synchronous APIs.
Thread Pool Bulkhead
- Separate thread pools per service.
- Prevents thread starvation.
- Best for async operations.
6. Production Request Flow
Client Request
|
v
API Gateway
|
Rate Limiter
|
Bulkhead
|
TimeLimiter
|
Retry
|
Circuit Breaker
|
Fallback
|
Downstream Service
7. Observability Stack
Monitoring Tools
- Prometheus
- Grafana
- OpenTelemetry
- Spring Boot Actuator
Key Metrics
- Latency (P50/P95/P99)
- Error Rate
- Throughput
- CPU Usage
- Memory Usage
- Thread Count
- Dependency Health
8. SRE Best Practices
- Define SLIs (Latency, Availability)
- Establish SLOs
- Perform RCA after incidents
- Conduct Disaster Recovery Drills
- Create Runbooks
- Automate Monitoring & Alerts
9. Gremlin Interview Questions
| Question |
Answer |
| What is Gremlin? |
Chaos Engineering platform used to simulate failures. |
| What is Blast Radius? |
Scope of impact of a chaos experiment. |
| What is Steady State? |
Normal system behavior baseline. |
| Latency vs Blackhole? |
Latency = Slow response, Blackhole = No response. |
| How to control Blast Radius? |
Target specific services, limit duration, use safeguards. |
| How to design a Chaos Experiment? |
Baseline ? Hypothesis ? Inject Failure ? Observe ? Improve. |
10. Real Production Impact
- Reduced MTTR (Mean Time To Recovery)
- Improved Service Reliability
- Prevented Cascading Failures
- Enhanced Observability
- Improved Auto-scaling Strategy
- Optimized Connection Pools
- Improved Deployment Confidence
11. Senior Engineer Interview One-Liner
Implemented Chaos Engineering using Gremlin to simulate network,
infrastructure, and application failures. Identified retry storms,
connection pool bottlenecks, and cascading failures. Introduced
Resilience4j patterns such as Circuit Breaker, Retry, Bulkhead,
Rate Limiter, TimeLimiter, and Fallback mechanisms. Enhanced
observability using Prometheus, Grafana, OpenTelemetry, and
Actuator dashboards, significantly improving system reliability
and reducing MTTR.