JMeter, Grafana, Prometheus Alerts & FMA Observability Guide
1. JMeter Keywords Cheat Sheet
| Keyword |
Meaning |
Usage / Explanation |
| Thread (User) | Virtual User | Simulates one real user. |
| Thread Group | Group of Users | Defines number of users and ramp-up. |
| Ramp-Up Period | User Start Time | Controls how quickly users are added. |
| Loop Count | Repetitions | Number of iterations per user. |
| Sampler | Request Element | Executes HTTP/JDBC/API requests. |
| HTTP Request | REST API Call | Invokes application endpoints. |
| Throughput | Requests per Second | System processing capacity. |
| Response Time | Total Response Duration | Time from request to complete response. |
| Latency | Time to First Byte | Initial response delay. |
| Connect Time | TCP Connection Time | Network connection setup duration. |
| TPS | Transactions per Second | Business transaction execution rate. |
| RPS | Requests per Second | API throughput measurement. |
| Error % | Failure Percentage | Failed requests percentage. |
| Assertions | Validation Checks | Verify response correctness. |
| Backend Listener | Metrics Export | Send metrics to Prometheus/Grafana. |
| CSV Data Set Config | External Test Data | Read test data from CSV. |
| Correlation | Dynamic Data Handling | Capture and reuse values. |
| Think Time | User Delay | Simulates realistic user behavior. |
| Distributed Testing | Multi-Machine Load | Scale load generation horizontally. |
2. Grafana Dashboard Design (FMA)
The purpose of the dashboard is to visualize system behavior before, during,
and after fault injection experiments.
| Panel |
PromQL Query |
Description |
Expected During Failure |
| Throughput (RPS) |
rate(http_server_requests_seconds_count[1m]) |
Requests processed per second |
Temporary drop |
| Response Time |
rate(http_server_requests_seconds_sum[1m]) / rate(http_server_requests_seconds_count[1m]) |
Average latency |
Spike observed |
| Error Rate |
rate(http_server_requests_seconds_count{status=~"5.."}[1m]) |
5xx failures |
Increase during outage |
| Pod Restart Count |
increase(kube_pod_container_status_restarts_total[5m]) |
Container restarts |
Increase after crash |
| CPU Usage |
rate(container_cpu_usage_seconds_total[1m]) |
CPU utilization |
May spike during recovery |
| Memory Usage |
container_memory_usage_bytes |
Memory consumption |
Observe JVM restart patterns |
| Pod Status |
kube_pod_status_phase |
Pod health state |
CrashLoopBackOff / Running |
Recommended Dashboard Layout
Row 1 : Throughput | Error Rate
Row 2 : Response Time
Row 3 : CPU Usage | Memory Usage
Row 4 : Pod Status | Pod Restarts
Row 5 : JVM Heap | GC Activity
Row 6 : HikariCP Connections | DB Latency
Row 7 : Retry Count | Circuit Breaker State
Best Practice:
Add Grafana annotations at the exact time Gremlin fault injection starts and ends.
This makes RCA and impact analysis much easier.
3. Prometheus Alert Rules
| Alert |
Expression |
Purpose |
| High Error Rate |
rate(http_server_requests_seconds_count{status=~"5.."}[1m]) > 1 |
Detect API failures |
| Pod Restart |
increase(kube_pod_container_status_restarts_total[5m]) > 0 |
Detect crashes |
| High Latency |
(rate(http_server_requests_seconds_sum[1m]) / rate(http_server_requests_seconds_count[1m])) > 1 |
Latency exceeds 1 second |
| Pod Down |
kube_pod_status_phase{phase!="Running"} == 1 |
Pod unavailable |
| High CPU |
rate(container_cpu_usage_seconds_total[5m]) > 0.8 |
CPU saturation |
| High Memory |
container_memory_usage_bytes > 80% |
Memory pressure |
Alert Channels
- Slack
- Email
- PagerDuty
- Microsoft Teams
- OpsGenie
4. FMA Observability Report
Scenario Name
Kill Main Process (Pod Crash Simulation)
Objective
Validate system resilience during unexpected pod termination and verify:
- Kubernetes auto-healing
- Traffic failover
- Recovery time
- Observability signals
- Customer impact
Test Setup
| Component | Value |
| Application | Data API |
| Environment | A3P Kubernetes |
| Load Tool | JMeter |
| Threads | 10 |
| Load Type | Continuous |
| Throughput | ~1.6 Requests/sec |
Failure Injection
kubectl exec POD_NAME -- /bin/bash -c "kill 1"
5. Timeline Analysis
| Event |
Time |
Observation |
| Load Started |
14:42:36 |
Steady state achieved |
| Pod Killed |
14:42:37 |
Main process terminated |
| Errors Observed |
14:42:37 - 14:42:40 |
Temporary failures returned |
| Pod Restarted |
~14:42:40 |
Kubernetes restarted container |
| Traffic Restored |
14:42:41 |
Requests processed normally |
| Recovery Completed |
14:42:41 |
System healthy again |
6. Observability Signals
| Category |
Observation |
| Throughput |
Brief drop during restart window |
| Error Rate |
Temporary increase in 5xx responses |
| Latency |
Slight spike during failover |
| Pod Restarts |
Restart counter increased by 1 |
| Logs |
Shutdown and startup sequences observed |
| Alerts |
Pod restart alert expected |
7. Detailed Impact Analysis
| Area |
Impact |
Severity |
| Availability |
No complete outage |
Low |
| User Requests |
Minor transient failures |
Low |
| Latency |
Temporary increase |
Low |
| Data Integrity |
No data loss observed |
None |
| Database |
No impact |
None |
| Recovery |
Automatic |
Excellent |
8. Additional Observations (Recommended)
| Observation |
Result |
| Traffic Failover |
Requests routed to healthy pods |
| Kubernetes Self-Healing |
Worked as expected |
| Load Balancer Behavior |
Successfully removed unhealthy endpoint |
| Readiness Probe |
Prevented traffic to restarting pod |
| Liveness Probe |
Detected process failure |
| Monitoring Coverage |
Metrics and logs captured incident |
| Alert Effectiveness |
Validated operational readiness |
| Customer Impact |
Minimal and short-lived |
9. Recovery Behavior
- Kubernetes detected container termination.
- Pod entered Terminating state.
- New container instance started automatically.
- Readiness probe delayed traffic until healthy.
- Traffic shifted back after successful startup.
- No manual intervention required.
10. Final Conclusion
| Evaluation Area |
Result |
Status |
| Pod Auto-Healing |
Successful |
PASS |
| Traffic Failover |
Successful |
PASS |
| Recovery Time |
~4 Seconds |
PASS |
| Observability |
Metrics, Logs, Alerts Captured |
PASS |
| Data Integrity |
No Data Loss |
PASS |
| Customer Experience |
Minor Transient Impact |
PASS |
| System Resilience |
High |
PASS |
Executive Summary:
The Kill Main Process chaos experiment successfully validated Kubernetes
self-healing capabilities. The application experienced a brief increase in
errors and a temporary throughput reduction during pod restart. Kubernetes
automatically restarted the failed pod and restored service within approximately
4 seconds. Traffic was successfully redirected to healthy instances, resulting
in minimal customer impact. Observability tooling accurately captured all
relevant signals, including pod restarts, error spikes, and recovery events.
Overall, the platform demonstrated strong resilience, effective failover
behavior, and production-grade recovery capabilities under single pod failure
conditions.