SEV-1 Production Incident Management

Complete SRE Guide: Triage • Metrics • Logs • Traces • Mitigation • Recovery • RCA

1. What is a SEV-1?

A SEV-1 is a critical production incident with major customer or business impact.

Key principle: During SEV-1, the first priority is to restore service and reduce customer impact. Perfect root-cause identification comes after stabilization.

Typical Examples

2. SEV-1 Incident Lifecycle

DETECT ↓ DECLARE SEV-1 ↓ TRIAGE ↓ ASSESS CUSTOMER/BUSINESS IMPACT ↓ CHECK METRICS ↓ CHECK LOGS ↓ CHECK TRACES ↓ CHECK DEPENDENCIES ↓ CHECK RECENT CHANGES ↓ FORM HYPOTHESIS ↓ MITIGATE ↓ VALIDATE RECOVERY ↓ MONITOR ↓ RCA ↓ CORRECTIVE ACTIONS

3. Step-by-Step SEV-1 Triage

Step 1 — Detect

Incidents may be detected through Prometheus, Grafana, Datadog, Dynatrace, New Relic, AppDynamics, CloudWatch, Azure Monitor, GCP Monitoring, or business alerts.

ALERT: Order Service Error Rate > 20%

Current Error Rate: 63%
Threshold: 5%
Severity: Critical
Environment: Production

Step 2 — Declare the Incident

Step 3 — Establish the Timeline

TimeEvent
13:45Deployment started
13:52Deployment completed
13:55Error rate increased
13:57Latency increased
14:00Customer complaints started
14:02SEV-1 alert triggered

Step 4 — Determine Business Impact

4. Golden Signals

The four core SRE signals are Latency, Traffic, Errors, and Saturation.

4.1 Latency

Check P50, P90, P95 and P99 rather than only averages.

Normal:
P95 = 250 ms

Incident:
P95 = 8.5 sec
P99 = 15 sec

4.2 Traffic

Normal traffic = 1000 requests/sec
Current traffic = 1000 requests/sec
Error rate = 70%

This suggests traffic is reaching the application but the application is failing.

4.3 Errors

500 Internal Server Error = 60%
502 Bad Gateway          = 10%
504 Gateway Timeout      = 15%

Break errors down by endpoint, service, region, pod, and dependency.

4.4 Saturation

CPU = 95%
Memory = 92%
DB connections = 100/100
Thread pool active = 200/200

5. Application Metrics for Java/Spring Boot

For Java microservices, inspect:

JVM MemoryGCCPU ThreadsThread PoolsHTTP Errors HTTP LatencyDB PoolKafka Lag
http_server_requests_seconds
jvm_memory_used_bytes
jvm_gc_pause_seconds
hikaricp_connections_active
hikaricp_connections_pending
process_cpu_usage
system_cpu_usage

6. Log Analysis

Do not randomly search millions of log lines. Narrow the investigation using timestamp, service, pod, request ID, trace ID, endpoint and exception.

2026-09-21 14:03:12 ERROR
OrderService
traceId=abc123
POST /orders
java.sql.SQLTransientConnectionException:
Connection is not available

Log Correlation

Gateway
   |
   | traceId=abc123
   ↓
Order Service
   |
   | traceId=abc123
   ↓
Payment Service
   |
   | traceId=abc123
   ↓
Database

Searching by the same trace ID can reconstruct the request path.

7. Distributed Tracing

Use OpenTelemetry, Jaeger, Zipkin, Dynatrace, Datadog, New Relic, or equivalent tooling.

POST /orders
      |
      | 20ms
      ↓
API Gateway
      |
      | 30ms
      ↓
Order Service
      |
      | 100ms
      ↓
Payment Service
      |
      | 7.8 sec
      ↓
SAP

If one downstream span consumes most of the request duration, investigate that dependency instead of blindly restarting the application.

Trace Data to Inspect

8. Dependency Analysis

                  ┌──────────┐
                  │   SAP    │
                  └────┬─────┘
                       │
                       ↓
┌────────┐      ┌─────────────┐      ┌──────────┐
│ Client │ ---> │ Order API   │ ---> │ Database │
└────────┘      └──────┬──────┘      └──────────┘
                       │
                       ↓
                  ┌──────────┐
                  │  Kafka   │
                  └──────────┘

Check:

9. Check Recent Changes

One of the highest-value triage questions is:

What changed immediately before the incident?
14:00 Deployment v2.8.1
14:05 Error rate increased

Compare:
v2.8.1  ← current version
v2.8.0  ← previous version

10. Mitigation During SEV-1

Remember: Mitigation is more important than perfect diagnosis during an active SEV-1.

Possible mitigation actions:

Action → Expected Result → Validation

ProblemActionExpected Result
High CPUScale replicasCPU per pod decreases
Bad deploymentRollbackErrors return toward baseline
Failing dependencyEnable circuit breakerApplication stops waiting on dependency
Regional outageFail overTraffic served by healthy region
Avoid: blindly deleting/restarting every pod. This can make an incident worse and destroy useful evidence.

11. Kubernetes SEV-1 Investigation

For a Kubernetes-hosted Java service:

kubectl get pods
kubectl describe pod <pod>
kubectl logs <pod>
kubectl logs <pod> --previous
kubectl get events
kubectl get deployment
kubectl describe deployment <deployment>

Inspect:

A pod being Running does not necessarily mean the application is healthy. Validate application-level health, metrics and business transactions.

12. Database Incident Analysis

DB connections = 100/100
DB latency = 5 sec
Application latency = 8 sec
HTTP 500 = 60%

Logs may show:

Connection is not available
Timeout after 30000ms

Investigate:

SHOW PROCESSLIST;

Possible mitigations depend on the evidence and change-control policy: stop a problematic workload, terminate a runaway query, scale/fail over the database, reduce traffic, or roll back an application change.

13. Kafka Incident Analysis

Consumer lag = 2,000,000
Consumer throughput = decreasing
Producer throughput = normal

Check:

Kafka message
     ↓
Consumer
     ↓
External API
     ↓
8-second timeout
     ↓
Consumer slows
     ↓
Kafka lag increases

14. SEV-1 Communication

Incident updates should be concise, factual and evidence-based.

SEV-1 UPDATE – 14:20 IST

Impact:
Approximately 60% of order transactions are failing.

Symptoms:
HTTP 5xx increased from 1% to 65%.
P95 latency increased from 250ms to 8 seconds.

Investigation:
Traces indicate elevated latency in the SAP integration.

Action:
Integration traffic has been reduced and the team is validating recovery.

Current status:
Error rate has decreased to 12%.

Next update:
14:30 IST.
Prefer: "Current evidence indicates SAP integration latency is contributing to the failures."
Avoid: "SAP is definitely broken" before evidence confirms it.

15. Incident Timeline Template

TimeEventEvidenceAction
14:00DeploymentCI/CDDeployment completed
14:04Errors increasePrometheusAlert triggered
14:06SEV-1 declaredIncident systemWar room opened
14:08P95 increasesGrafanaInvestigation started
14:12DB connections exhaustedMetrics/logsDB investigation
14:15Deployment suspectedDeployment historyRollback approved
14:18Rollback completeKubernetesMonitoring
14:22Error rate normalPrometheusRecovery confirmed

16. Root Cause Analysis

5 Whys Example

  1. Why did orders fail? Order Service returned HTTP 500.
  2. Why? Database connections were exhausted.
  3. Why? Queries were taking much longer than normal.
  4. Why? A new query caused a full table scan.
  5. Why? The required database index was missing from the migration.

Separate These Concepts

ConceptExample
TriggerDeployment v2.8.1
Root CauseMissing database index
Contributing FactorNo production-like query performance test
Detection GapNo database query-latency alert
Recovery GapRollback required manual approval

Corrective Actions

17. FMA / Alert Mapping

In an FMA-oriented SRE environment, examples include:

APPLICATION_ERROR HIGH_LATENCY SAP_TIMEOUT DATABASE_CONNECTION_FAILURE KAFKA_LAG POD_CRASH HIGH_CPU HIGH_MEMORY DISK_FULL NETWORK_FAILURE CERTIFICATE_EXPIRY DEPLOYMENT_FAILURE

Example FMA Chain

Customer
   ↓
Application
   ↓
Integration
   ↓
SAP

SAP latency
      ↓
Integration timeout
      ↓
HTTP 504
      ↓
Application error
      ↓
Customer transaction failure

Possible FMA chain:
SAP_TIMEOUT
     ↓
HIGH_LATENCY
     ↓
APPLICATION_ERROR

Always distinguish the originating failure from downstream symptoms.

18. SEV-1 Decision Tree

SEV-1 ALERT | ↓ Is customer impact confirmed? | +---- NO → Continue investigation | +---- YES ↓ Declare SEV-1 ↓ Check Golden Signals ↓ +------+-------+-------+ | | | Errors Latency Traffic | | | +------+-------+-------+ ↓ Check Saturation ↓ Check Logs ↓ Check Traces ↓ Check Dependencies ↓ Check Recent Changes ↓ Form Hypothesis ↓ Mitigate ↓ Validate / \ Recovered Not recovered | | ↓ ↓ Monitor Continue Triage | ↓ RCA | ↓ Corrective Actions

19. Key Questions During Triage

Impact

Time

Metrics

Logs & Traces

Infrastructure & Dependencies

20. Interview Answer — “How Do You Handle a SEV-1?”

Sample Answer

For a SEV-1 production incident, my first priority is to stabilize the system and reduce customer impact rather than immediately trying to find the complete root cause.

I first confirm the impact and declare the incident according to the organization's severity criteria. I establish an Incident Commander and technical owners, open the incident bridge, and start a timeline.

For technical triage, I look at the four golden signals: traffic, latency, errors and saturation. In a Java microservices environment, I additionally check JVM memory, GC, thread pools, database connection pools, Kafka lag and downstream dependency metrics.

Once I identify the affected service, I correlate metrics with application logs using timestamps, request IDs and trace IDs. I then use distributed tracing to identify where the request is failing or spending most of its time.

Next, I check dependencies such as databases, Kafka, Redis, external APIs and SAP, and compare the incident timeline against recent deployments, configuration changes, database migrations and infrastructure changes.

I formulate hypotheses and validate them using metrics, logs and traces rather than assumptions. If a recent deployment is strongly correlated and rollback is safe, I may roll it back. Depending on the failure, other mitigations could include scaling, disabling a feature flag, failing over a dependency or isolating unhealthy instances.

After mitigation, I validate recovery using metrics, logs, traces and actual business transactions. I continue monitoring to make sure the problem does not recur.

Once the service is stable, I conduct RCA using the incident timeline and evidence. I distinguish the trigger, root cause, contributing factors and detection gaps, and create corrective actions covering code, infrastructure, monitoring, testing and operational processes.

21. SRE Mental Model

CUSTOMER IMPACT ↓ BUSINESS IMPACT ↓ SERVICE IMPACT ↓ METRICS ↓ LOGS ↓ TRACES ↓ DEPENDENCIES ↓ RECENT CHANGES ↓ HYPOTHESIS ↓ MITIGATION ↓ VALIDATION ↓ RCA ↓ PREVENTION
Golden Rule: Stabilize first. Diagnose with evidence. Communicate continuously. Validate recovery. Then perform deep RCA.