Chaos Engineering & Resilience Engineering Notes

Interview Summary: Chaos Engineering helps uncover hidden production failures before customers experience them. The goal is to improve system reliability, resiliency, observability, and recovery mechanisms.

1. Why Chaos Engineering?


2. Failure Scenarios Tested

Network-Level Failures

Infrastructure-Level Failures

Application-Level Failures


3. Observed Problems


4. Resilience Patterns

Pattern Purpose
Circuit Breaker Stops repeated calls to failing services
Retry Handles transient failures
Exponential Backoff Controls retry frequency
Timeout Prevents thread blocking
Bulkhead Isolates resources
Rate Limiter Controls traffic volume
Fallback Provides alternate response
Cache Reduces repeated calls
TimeLimiter Limits execution time
Graceful Degradation Keeps critical services operational

5. Bulkhead Types

Semaphore Bulkhead

Thread Pool Bulkhead


6. Production Request Flow

Client Request | v API Gateway | Rate Limiter | Bulkhead | TimeLimiter | Retry | Circuit Breaker | Fallback | Downstream Service

7. Observability Stack

Monitoring Tools

Key Metrics


8. SRE Best Practices


9. Gremlin Interview Questions

Question Answer
What is Gremlin? Chaos Engineering platform used to simulate failures.
What is Blast Radius? Scope of impact of a chaos experiment.
What is Steady State? Normal system behavior baseline.
Latency vs Blackhole? Latency = Slow response, Blackhole = No response.
How to control Blast Radius? Target specific services, limit duration, use safeguards.
How to design a Chaos Experiment? Baseline ? Hypothesis ? Inject Failure ? Observe ? Improve.

10. Real Production Impact


11. Senior Engineer Interview One-Liner

Implemented Chaos Engineering using Gremlin to simulate network, infrastructure, and application failures. Identified retry storms, connection pool bottlenecks, and cascading failures. Introduced Resilience4j patterns such as Circuit Breaker, Retry, Bulkhead, Rate Limiter, TimeLimiter, and Fallback mechanisms. Enhanced observability using Prometheus, Grafana, OpenTelemetry, and Actuator dashboards, significantly improving system reliability and reducing MTTR.