SRE, RCA, Gremlin & Network Failure Scenarios

Senior Java / Spring Boot / Microservices / Kubernetes / Cloud Interview Guide

Practical interview points for reliability assessment, incident analysis, chaos engineering and network failures.

SRARCAGremlin Chaos EngineeringBlackholeLatencyPacket Loss SREKubernetes

Quick Navigation

1. SRA — Service / Site Reliability Assessment

Interview note: In reliability/operations discussions, SRA is commonly used for a Service Reliability Assessment or Site Reliability Assessment. The exact expansion can vary by organization. Always align with the terminology used by the interviewer/company.

One-line definition: An SRA is a structured assessment of whether a service is designed and operated to meet reliability, availability, performance, security and operational requirements.

What an SRA Typically Checks

AreaWhat to CheckInterview Point
AvailabilityRedundancy, replicas, failoverCan the service continue when an instance/Pod/Node fails?
ScalabilityHorizontal scaling, HPA, capacityCan the service handle traffic growth?
PerformanceLatency, throughput, resource usageAre response-time targets defined and measured?
ResilienceTimeouts, retries, circuit breakers, graceful degradationWhat happens when dependencies fail?
ObservabilityLogs, metrics, traces, alertsCan we detect and diagnose an incident?
DeploymentRolling/canary/blue-green deploymentCan changes be released and rolled back safely?
SecurityIAM/RBAC, secrets, TLS, vulnerability controlsAre access and sensitive data handled securely?
Disaster RecoveryBackup, restore, RTO, RPOCan service/data be recovered after a major failure?
OperationsRunbooks, on-call, escalation, ownershipDoes the team know how to respond to incidents?
Strong interview answer: “Before production, I would assess availability, scalability, resilience, observability, deployment strategy, security and disaster recovery. I would also verify SLOs, alerting, runbooks and failure recovery through controlled testing.”

2. RCA — Root Cause Analysis

Full form: RCA = Root Cause Analysis

One-line definition: RCA is the structured process of identifying the underlying cause of an incident rather than only treating its symptoms.

RCA vs Symptom vs Root Cause

LevelExample
SymptomUsers receive HTTP 503 responses.
Immediate CauseAll application Pods are failing readiness checks.
Deeper CauseApplication cannot connect to the database.
Root CauseA database connection-pool configuration change exhausted available DB connections after deployment.
Corrective ActionFix pool configuration and add connection saturation monitoring.
Preventive ActionAdd deployment validation, load testing and alerting before reaching saturation.

RCA Process

Incident ↓ Detect & Declare ↓ Collect Evidence ↓ Establish Timeline ↓ Identify Symptoms ↓ Trace Dependencies ↓ Test Hypotheses ↓ Find Root Cause ↓ Immediate Mitigation ↓ Corrective / Preventive Actions ↓ Validate & Close

Useful RCA Techniques

TechniqueInterview Explanation
5 WhysRepeatedly ask why the failure happened until reaching an actionable underlying cause.
Fishbone / IshikawaClassify possible causes across areas such as people, process, technology, environment and dependencies.
Timeline AnalysisCorrelate deployments, configuration changes, alerts and failures by time.
Change CorrelationCheck whether a recent code/config/infrastructure change correlates with the incident.
Dependency AnalysisTrace database, cache, API, DNS, network and messaging dependencies.
Important: A good RCA should avoid blame. Focus on system conditions, contributing factors, detection gaps and prevention.

3. Gremlin — Chaos Engineering Platform

One-line definition: Gremlin is a chaos engineering platform used to intentionally introduce controlled failures and validate whether systems can withstand them.

Why Use Gremlin?

Use CaseWhat You TestExpected Learning
CPU stressHigh CPU consumptionDoes autoscaling work? Do latency and alerts behave correctly?
Memory stressMemory pressureDoes the application handle memory pressure safely?
Network latencyArtificial delayAre timeouts, retries and circuit breakers effective?
Packet lossDrop network packetsCan clients recover from unreliable network communication?
Network blackholeBlock network communicationDoes the service fail gracefully when a dependency becomes unreachable?
Shutdown / process failureTerminate a process/instanceDoes Kubernetes/cloud self-heal correctly?
Disk stressDisk I/O or capacity pressureDoes the application remain stable under storage pressure?
Senior interview line: “We use chaos engineering to validate resilience assumptions before a real production failure validates them for us.”

Gremlin Experiment Lifecycle

Define Hypothesis ↓ Define Blast Radius ↓ Check Safety / Abort Conditions ↓ Run Controlled Experiment ↓ Observe Metrics / Logs / Traces ↓ Validate SLOs ↓ Stop / Roll Back ↓ Document Findings ↓ Improve System
Safety principle: Chaos experiments should be controlled, authorized and limited in blast radius. Start in non-production, then progress to carefully controlled production experiments only when appropriate.

4. Chaos Engineering — Interview Perspective

Chaos Engineering: The practice of deliberately injecting controlled failures into a system to discover weaknesses and improve resilience.

Normal TestingChaos Testing
Tests expected behavior.Tests behavior under failure.
“Does API return 200?”“What happens if the database is unreachable?”
Usually deterministic scenarios.Introduces realistic infrastructure/dependency failures.
Validates functionality.Validates resilience and recovery.

Common Chaos Experiments

5. Network Blackhole

One-line definition: A network blackhole intentionally or unintentionally causes packets destined for a target to be dropped or routed nowhere, making the destination effectively unreachable.

Example

Order Service │ │ request ▼ Payment Service X BLACKHOLE │ └── Packets disappear / cannot reach destination

What Does the Application Experience?

LayerPossible Behavior
NetworkPackets are dropped or never reach destination.
TCPConnection attempts may time out or existing connections may fail.
ApplicationRequests wait until timeout unless bounded by application settings.
MicroserviceThread/connection resources can become exhausted if timeouts are poorly configured.
UserSlow response, error, timeout or degraded functionality.

Why Test a Blackhole?

Interview warning: Unlimited retries against a blackholed dependency can amplify an incident. Use bounded timeouts, controlled retries and circuit breaking.

6. Network Latency

One-line definition: Network latency is the time taken for data to travel between endpoints and/or for a request to receive a response.

Example

Client → API Gateway → Order Service → Payment Service Normal: 20 ms → 30 ms → 40 ms Injected latency: 20 ms → 30 ms → 2,000 ms Result: Overall API response becomes slow ↓ Timeouts may occur ↓ Retries may start ↓ Resources can become saturated

Causes

CauseExample
Geographic distanceService deployed in a distant region.
Network congestionHigh traffic or constrained network path.
Cross-zone/region communicationMicroservices deployed across different zones/regions.
Proxy/load balancerAdditional network hops.
DNS delaysSlow name resolution.
Dependency slownessDatabase or downstream API takes longer.

What Should a Senior Java Developer Check?

7. Packet Loss

One-line definition: Packet loss occurs when network packets fail to reach their destination, requiring retransmission or causing communication failure.

Packet Loss vs Latency vs Blackhole

FailureMeaningTypical Effect
LatencyPackets arrive but take longer.Slow requests/timeouts.
Packet LossSome packets do not arrive.Retransmissions, degraded throughput, intermittent errors.
BlackholeTraffic to a target is effectively dropped/unreachable.Requests may consistently time out.

Impact on Microservices

Key distinction: Packet loss does not necessarily mean the entire service is unavailable. A small amount of loss can create intermittent slowness/failures, while severe loss can make communication effectively unusable.

8. Real Interview Scenario — Payment Service Network Failure

Scenario

Suppose Order Service calls Payment Service. A chaos experiment introduces 2 seconds of network latency and 10% packet loss.

Order Service ↓ Payment Service ↓ [2 sec latency + 10% packet loss] ↓ Payment response becomes slow/intermittent Without resilience: Request waits ↓ Thread/connection held ↓ Timeout ↓ Retry ↓ More traffic ↓ Resource exhaustion ↓ Cascading failure With resilience: Timeout ↓ Bounded retry + exponential backoff ↓ Circuit breaker ↓ Fallback / graceful degradation ↓ Alert + trace + SLO monitoring

What Should Be Configured?

ControlPurpose
Connect TimeoutLimits time spent establishing a connection.
Read/Response TimeoutLimits waiting for the dependency response.
Bounded RetriesRetries transient failures without creating an infinite retry storm.
Exponential BackoffSpaces retries progressively.
JitterRandomizes retry timing to avoid synchronized retry spikes.
Circuit BreakerStops repeatedly calling a failing dependency.
BulkheadLimits resource consumption by one dependency/workload.
FallbackProvides controlled degraded behavior where business logic permits.
ObservabilityMeasures latency, errors, saturation and dependency health.

9. Observability — What to Monitor During an Experiment

SignalWhat to Look For
MetricsRequest rate, error rate, latency, CPU, memory, connections, queue depth.
LogsTimeouts, connection errors, retry messages, circuit-breaker transitions.
TracesWhich downstream call is consuming most of the request time.
KubernetesPod restarts, readiness failures, endpoint changes, CPU/memory pressure.
NetworkLatency, retransmissions, packet drops, connection failures.
SLOWhether availability and latency objectives remain within target.

Golden Metrics

Latency — How long requests take.

Traffic — How much demand the service receives.

Errors — How many requests fail.

Saturation — How close resources are to their limits.

Useful Percentiles

MetricInterview Meaning
AverageOverall mean, but can hide tail latency.
p50Median request latency.
p9595% of requests are at or below this latency.
p9999% of requests are at or below this latency; useful for tail behavior.

10. Senior Interview Q&A

Q1. What is RCA?RCA stands for Root Cause Analysis. It identifies the underlying cause of an incident and defines corrective and preventive actions so the problem is less likely to recur.
Q2. What is SRA?In reliability discussions, SRA commonly means Service Reliability Assessment or Site Reliability Assessment. It evaluates whether a service is operationally ready and resilient across availability, scalability, observability, security, deployment and recovery.
Q3. What is Gremlin?Gremlin is a chaos engineering platform used to run controlled failure experiments such as CPU stress, network latency, packet loss, blackholes and instance/process failures.
Q4. Why introduce network latency?To verify that timeout, retry, circuit-breaker, connection-pool and graceful-degradation mechanisms work when downstream responses become slow.
Q5. What is a network blackhole?A blackhole is a condition where traffic to a target is effectively dropped or has no usable route, causing communication to fail or time out.
Q6. What is packet loss?Packet loss means some network packets fail to reach their destination. It can cause retransmissions, degraded throughput, intermittent errors and increased latency.
Q7. Difference between latency and packet loss?Latency means packets take longer to arrive; packet loss means packets do not arrive. Packet loss can indirectly increase latency because protocols may retransmit lost data.
Q8. What happens if a downstream service is blackholed?If timeouts are not bounded, application threads and connections may remain occupied, retries can amplify traffic, and the failure can cascade. Proper timeouts, bounded retries, circuit breakers and bulkheads prevent this.
Q9. How would you test Kubernetes resilience?I would use controlled experiments such as Pod termination, Node failure, network latency, packet loss or dependency blackholing, while monitoring SLOs, metrics, logs, traces, Pod readiness and recovery behavior.
Q10. What is the difference between monitoring and chaos engineering?Monitoring tells us that a system is behaving badly; chaos engineering deliberately creates controlled failures to verify how the system behaves and recovers.
Q11. What is a good RCA?A good RCA establishes a timeline, identifies contributing factors and root cause, explains impact, documents mitigation, and creates measurable corrective/preventive actions without assigning blame.
Q12. What is a retry storm?A retry storm occurs when many clients repeatedly retry a failing or slow dependency, increasing load and potentially turning a partial failure into a cascading outage.
Q13. Why use exponential backoff and jitter?Exponential backoff reduces retry frequency, while jitter prevents many clients from retrying at exactly the same time.
Q14. What is a circuit breaker?A circuit breaker temporarily stops calls to a failing dependency after a threshold is reached, allowing the dependency and caller to recover and preventing repeated failure amplification.

11. 60-Second Senior Interview Answer

“For production reliability, I look at reliability assessment, incident management and controlled resilience testing. An SRA helps evaluate whether a service is ready from availability, scalability, resilience, observability, security and disaster-recovery perspectives. If an incident occurs, I perform RCA by establishing the timeline, correlating deployments and dependency behavior, identifying the root cause, and defining corrective and preventive actions. For resilience validation, we can use chaos engineering tools such as Gremlin to introduce controlled failures like Pod termination, CPU or memory stress, network latency, packet loss and network blackholes. For example, if Payment Service becomes unreachable, I expect the Order Service to have bounded connection/read timeouts, limited retries with exponential backoff and jitter, circuit breakers and appropriate fallback behavior. During the experiment I monitor latency, error rate, traffic, saturation, logs, traces, Kubernetes health and SLOs. The goal is not simply to prove that the system fails, but to prove that it fails gracefully, recovers automatically where possible, and gives the operations team enough observability to diagnose the issue.”

12. Final Memory Map

SRA ↓ Assess Reliability Readiness ↓ SLI / SLO / SLA ↓ Observability ↓ Chaos Engineering ↓ Gremlin ↓ Inject Controlled Failure ├── CPU Stress ├── Memory Stress ├── Pod / Process Failure ├── Network Latency ├── Packet Loss └── Network Blackhole ↓ Observe ├── Metrics ├── Logs ├── Traces └── Kubernetes Events ↓ RCA ↓ Root Cause + Contributing Factors ↓ Mitigation ↓ Corrective / Preventive Actions ↓ Retest Resilience
Five lines to memorize:
1. SRA assesses whether a service is reliable and operationally ready.
2. RCA finds the underlying cause of an incident and prevents recurrence.
3. Gremlin is used for controlled chaos/resilience experiments.
4. Latency slows communication; packet loss drops packets; a blackhole makes a target effectively unreachable.
5. Timeouts + bounded retries + backoff/jitter + circuit breakers + observability prevent network failures from becoming cascading failures.