Complete SRE Guide: Triage • Metrics • Logs • Traces • Mitigation • Recovery • RCA
A SEV-1 is a critical production incident with major customer or business impact.
Incidents may be detected through Prometheus, Grafana, Datadog, Dynatrace, New Relic, AppDynamics, CloudWatch, Azure Monitor, GCP Monitoring, or business alerts.
ALERT: Order Service Error Rate > 20% Current Error Rate: 63% Threshold: 5% Severity: Critical Environment: Production
| Time | Event |
|---|---|
| 13:45 | Deployment started |
| 13:52 | Deployment completed |
| 13:55 | Error rate increased |
| 13:57 | Latency increased |
| 14:00 | Customer complaints started |
| 14:02 | SEV-1 alert triggered |
The four core SRE signals are Latency, Traffic, Errors, and Saturation.
Check P50, P90, P95 and P99 rather than only averages.
Normal: P95 = 250 ms Incident: P95 = 8.5 sec P99 = 15 sec
Normal traffic = 1000 requests/sec Current traffic = 1000 requests/sec Error rate = 70%
This suggests traffic is reaching the application but the application is failing.
500 Internal Server Error = 60% 502 Bad Gateway = 10% 504 Gateway Timeout = 15%
Break errors down by endpoint, service, region, pod, and dependency.
CPU = 95% Memory = 92% DB connections = 100/100 Thread pool active = 200/200
For Java microservices, inspect:
http_server_requests_seconds jvm_memory_used_bytes jvm_gc_pause_seconds hikaricp_connections_active hikaricp_connections_pending process_cpu_usage system_cpu_usage
Do not randomly search millions of log lines. Narrow the investigation using timestamp, service, pod, request ID, trace ID, endpoint and exception.
2026-09-21 14:03:12 ERROR OrderService traceId=abc123 POST /orders java.sql.SQLTransientConnectionException: Connection is not available
Gateway | | traceId=abc123 ↓ Order Service | | traceId=abc123 ↓ Payment Service | | traceId=abc123 ↓ Database
Searching by the same trace ID can reconstruct the request path.
Use OpenTelemetry, Jaeger, Zipkin, Dynatrace, Datadog, New Relic, or equivalent tooling.
POST /orders
|
| 20ms
↓
API Gateway
|
| 30ms
↓
Order Service
|
| 100ms
↓
Payment Service
|
| 7.8 sec
↓
SAP
If one downstream span consumes most of the request duration, investigate that dependency instead of blindly restarting the application.
┌──────────┐
│ SAP │
└────┬─────┘
│
↓
┌────────┐ ┌─────────────┐ ┌──────────┐
│ Client │ ---> │ Order API │ ---> │ Database │
└────────┘ └──────┬──────┘ └──────────┘
│
↓
┌──────────┐
│ Kafka │
└──────────┘
Check:
One of the highest-value triage questions is:
14:00 Deployment v2.8.1 14:05 Error rate increased Compare: v2.8.1 ← current version v2.8.0 ← previous version
Possible mitigation actions:
| Problem | Action | Expected Result |
|---|---|---|
| High CPU | Scale replicas | CPU per pod decreases |
| Bad deployment | Rollback | Errors return toward baseline |
| Failing dependency | Enable circuit breaker | Application stops waiting on dependency |
| Regional outage | Fail over | Traffic served by healthy region |
For a Kubernetes-hosted Java service:
kubectl get pods kubectl describe pod <pod> kubectl logs <pod> kubectl logs <pod> --previous kubectl get events kubectl get deployment kubectl describe deployment <deployment>
Inspect:
Running does not necessarily mean the application is healthy. Validate application-level health, metrics and business transactions.DB connections = 100/100 DB latency = 5 sec Application latency = 8 sec HTTP 500 = 60%
Logs may show:
Connection is not available Timeout after 30000ms
Investigate:
SHOW PROCESSLIST;
Possible mitigations depend on the evidence and change-control policy: stop a problematic workload, terminate a runaway query, scale/fail over the database, reduce traffic, or roll back an application change.
Consumer lag = 2,000,000 Consumer throughput = decreasing Producer throughput = normal
Check:
Kafka message
↓
Consumer
↓
External API
↓
8-second timeout
↓
Consumer slows
↓
Kafka lag increases
Incident updates should be concise, factual and evidence-based.
SEV-1 UPDATE – 14:20 IST Impact: Approximately 60% of order transactions are failing. Symptoms: HTTP 5xx increased from 1% to 65%. P95 latency increased from 250ms to 8 seconds. Investigation: Traces indicate elevated latency in the SAP integration. Action: Integration traffic has been reduced and the team is validating recovery. Current status: Error rate has decreased to 12%. Next update: 14:30 IST.
| Time | Event | Evidence | Action |
|---|---|---|---|
| 14:00 | Deployment | CI/CD | Deployment completed |
| 14:04 | Errors increase | Prometheus | Alert triggered |
| 14:06 | SEV-1 declared | Incident system | War room opened |
| 14:08 | P95 increases | Grafana | Investigation started |
| 14:12 | DB connections exhausted | Metrics/logs | DB investigation |
| 14:15 | Deployment suspected | Deployment history | Rollback approved |
| 14:18 | Rollback complete | Kubernetes | Monitoring |
| 14:22 | Error rate normal | Prometheus | Recovery confirmed |
| Concept | Example |
|---|---|
| Trigger | Deployment v2.8.1 |
| Root Cause | Missing database index |
| Contributing Factor | No production-like query performance test |
| Detection Gap | No database query-latency alert |
| Recovery Gap | Rollback required manual approval |
In an FMA-oriented SRE environment, examples include:
Customer
↓
Application
↓
Integration
↓
SAP
SAP latency
↓
Integration timeout
↓
HTTP 504
↓
Application error
↓
Customer transaction failure
Possible FMA chain:
SAP_TIMEOUT
↓
HIGH_LATENCY
↓
APPLICATION_ERROR
Always distinguish the originating failure from downstream symptoms.
Sample Answer
For a SEV-1 production incident, my first priority is to stabilize the system and reduce customer impact rather than immediately trying to find the complete root cause.
I first confirm the impact and declare the incident according to the organization's severity criteria. I establish an Incident Commander and technical owners, open the incident bridge, and start a timeline.
For technical triage, I look at the four golden signals: traffic, latency, errors and saturation. In a Java microservices environment, I additionally check JVM memory, GC, thread pools, database connection pools, Kafka lag and downstream dependency metrics.
Once I identify the affected service, I correlate metrics with application logs using timestamps, request IDs and trace IDs. I then use distributed tracing to identify where the request is failing or spending most of its time.
Next, I check dependencies such as databases, Kafka, Redis, external APIs and SAP, and compare the incident timeline against recent deployments, configuration changes, database migrations and infrastructure changes.
I formulate hypotheses and validate them using metrics, logs and traces rather than assumptions. If a recent deployment is strongly correlated and rollback is safe, I may roll it back. Depending on the failure, other mitigations could include scaling, disabling a feature flag, failing over a dependency or isolating unhealthy instances.
After mitigation, I validate recovery using metrics, logs, traces and actual business transactions. I continue monitoring to make sure the problem does not recur.
Once the service is stable, I conduct RCA using the incident timeline and evidence. I distinguish the trigger, root cause, contributing factors and detection gaps, and create corrective actions covering code, infrastructure, monitoring, testing and operational processes.