JMeter, Grafana, Prometheus Alerts & FMA Observability Guide

1. JMeter Keywords Cheat Sheet

Keyword Meaning Usage / Explanation
Thread (User)Virtual UserSimulates one real user.
Thread GroupGroup of UsersDefines number of users and ramp-up.
Ramp-Up PeriodUser Start TimeControls how quickly users are added.
Loop CountRepetitionsNumber of iterations per user.
SamplerRequest ElementExecutes HTTP/JDBC/API requests.
HTTP RequestREST API CallInvokes application endpoints.
ThroughputRequests per SecondSystem processing capacity.
Response TimeTotal Response DurationTime from request to complete response.
LatencyTime to First ByteInitial response delay.
Connect TimeTCP Connection TimeNetwork connection setup duration.
TPSTransactions per SecondBusiness transaction execution rate.
RPSRequests per SecondAPI throughput measurement.
Error %Failure PercentageFailed requests percentage.
AssertionsValidation ChecksVerify response correctness.
Backend ListenerMetrics ExportSend metrics to Prometheus/Grafana.
CSV Data Set ConfigExternal Test DataRead test data from CSV.
CorrelationDynamic Data HandlingCapture and reuse values.
Think TimeUser DelaySimulates realistic user behavior.
Distributed TestingMulti-Machine LoadScale load generation horizontally.

2. Grafana Dashboard Design (FMA)

The purpose of the dashboard is to visualize system behavior before, during, and after fault injection experiments.

Panel PromQL Query Description Expected During Failure
Throughput (RPS) rate(http_server_requests_seconds_count[1m]) Requests processed per second Temporary drop
Response Time rate(http_server_requests_seconds_sum[1m]) / rate(http_server_requests_seconds_count[1m]) Average latency Spike observed
Error Rate rate(http_server_requests_seconds_count{status=~"5.."}[1m]) 5xx failures Increase during outage
Pod Restart Count increase(kube_pod_container_status_restarts_total[5m]) Container restarts Increase after crash
CPU Usage rate(container_cpu_usage_seconds_total[1m]) CPU utilization May spike during recovery
Memory Usage container_memory_usage_bytes Memory consumption Observe JVM restart patterns
Pod Status kube_pod_status_phase Pod health state CrashLoopBackOff / Running

Recommended Dashboard Layout

Row 1 : Throughput | Error Rate

Row 2 : Response Time

Row 3 : CPU Usage | Memory Usage

Row 4 : Pod Status | Pod Restarts

Row 5 : JVM Heap | GC Activity

Row 6 : HikariCP Connections | DB Latency

Row 7 : Retry Count | Circuit Breaker State
Best Practice: Add Grafana annotations at the exact time Gremlin fault injection starts and ends. This makes RCA and impact analysis much easier.

3. Prometheus Alert Rules

Alert Expression Purpose
High Error Rate rate(http_server_requests_seconds_count{status=~"5.."}[1m]) > 1 Detect API failures
Pod Restart increase(kube_pod_container_status_restarts_total[5m]) > 0 Detect crashes
High Latency (rate(http_server_requests_seconds_sum[1m]) / rate(http_server_requests_seconds_count[1m])) > 1 Latency exceeds 1 second
Pod Down kube_pod_status_phase{phase!="Running"} == 1 Pod unavailable
High CPU rate(container_cpu_usage_seconds_total[5m]) > 0.8 CPU saturation
High Memory container_memory_usage_bytes > 80% Memory pressure

Alert Channels

4. FMA Observability Report

Scenario Name

Kill Main Process (Pod Crash Simulation)

Objective

Validate system resilience during unexpected pod termination and verify:

Test Setup

ComponentValue
ApplicationData API
EnvironmentA3P Kubernetes
Load ToolJMeter
Threads10
Load TypeContinuous
Throughput~1.6 Requests/sec

Failure Injection

kubectl exec POD_NAME -- /bin/bash -c "kill 1"

5. Timeline Analysis

Event Time Observation
Load Started 14:42:36 Steady state achieved
Pod Killed 14:42:37 Main process terminated
Errors Observed 14:42:37 - 14:42:40 Temporary failures returned
Pod Restarted ~14:42:40 Kubernetes restarted container
Traffic Restored 14:42:41 Requests processed normally
Recovery Completed 14:42:41 System healthy again

6. Observability Signals

Category Observation
Throughput Brief drop during restart window
Error Rate Temporary increase in 5xx responses
Latency Slight spike during failover
Pod Restarts Restart counter increased by 1
Logs Shutdown and startup sequences observed
Alerts Pod restart alert expected

7. Detailed Impact Analysis

Area Impact Severity
Availability No complete outage Low
User Requests Minor transient failures Low
Latency Temporary increase Low
Data Integrity No data loss observed None
Database No impact None
Recovery Automatic Excellent

8. Additional Observations (Recommended)

Observation Result
Traffic Failover Requests routed to healthy pods
Kubernetes Self-Healing Worked as expected
Load Balancer Behavior Successfully removed unhealthy endpoint
Readiness Probe Prevented traffic to restarting pod
Liveness Probe Detected process failure
Monitoring Coverage Metrics and logs captured incident
Alert Effectiveness Validated operational readiness
Customer Impact Minimal and short-lived

9. Recovery Behavior

10. Final Conclusion

Evaluation Area Result Status
Pod Auto-Healing Successful PASS
Traffic Failover Successful PASS
Recovery Time ~4 Seconds PASS
Observability Metrics, Logs, Alerts Captured PASS
Data Integrity No Data Loss PASS
Customer Experience Minor Transient Impact PASS
System Resilience High PASS
Executive Summary: The Kill Main Process chaos experiment successfully validated Kubernetes self-healing capabilities. The application experienced a brief increase in errors and a temporary throughput reduction during pod restart. Kubernetes automatically restarted the failed pod and restored service within approximately 4 seconds. Traffic was successfully redirected to healthy instances, resulting in minimal customer impact. Observability tooling accurately captured all relevant signals, including pod restarts, error spikes, and recovery events. Overall, the platform demonstrated strong resilience, effective failover behavior, and production-grade recovery capabilities under single pod failure conditions.