Prometheus, Grafana, PushGateway & Chaos Engineering
1. PushGateway Configuration
management:
metrics:
export:
pushgateway:
enabled: true
base-url: http://pushgateway:9091
job: ref-legacy-api-container
push-rate: 10s
shutdown-operation: DELETE
grouping-key:
instance: ${HOSTNAME}
What This Configuration Means
- Metrics pushed every 10 seconds
- Job Name = ref-legacy-api-container
- Instance = Current Pod Hostname
- Metrics deleted on shutdown
Best Practice:
Always configure shutdown-operation: DELETE to avoid stale metrics.
2. PushGateway Deployment
Docker Setup
docker run -d \
-p 9091:9091 \
--name pushgateway \
prom/pushgateway
Access:
http://localhost:9091
Kubernetes Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: pushgateway
spec:
replicas: 1
selector:
matchLabels:
app: pushgateway
template:
metadata:
labels:
app: pushgateway
spec:
containers:
- name: pushgateway
image: prom/pushgateway
ports:
- containerPort: 9091
3. Prometheus Configuration
scrape_configs:
- job_name: 'pushgateway'
honor_labels: true
static_configs:
- targets:
- pushgateway:9091
Why honor_labels: true?
Prevents Prometheus from overwriting labels pushed by PushGateway.
4. Monitoring Architecture
Spring Boot Application
|
| Push Metrics
V
PushGateway
|
| Scrape
V
Prometheus
|
V
Grafana
5. Chaos Engineering Architecture
JMeter (Load Testing)
|
V
Spring Boot Application
|
V
Kubernetes Cluster
|
V
Gremlin Chaos Experiments
|
V
Prometheus
|
V
Grafana Dashboard
6. Grafana Dashboard Panels
| Panel |
Purpose |
Interpretation |
| Latency |
Response Time |
Spike indicates performance degradation |
| Throughput |
Requests per second |
Drop indicates load handling issue |
| Error Rate |
Failed Requests |
Increase indicates failures |
| CPU |
Resource Consumption |
High CPU means overload |
| Memory |
Heap Usage |
Continuous increase may indicate leak |
| Pod Health |
Restart Count |
Frequent restarts indicate instability |
7. RCA Timeline Example
| Time |
Event |
| 4:52 PM |
Gremlin Blackhole Started |
| 4:55 PM |
Latency Increased |
| 4:58 PM |
Chaos Interrupted |
| 4:59 PM |
System Recovery Observed |
8. Root Cause Analysis
Primary Cause:
- Gremlin experiment terminated unexpectedly
- Session timeout
- VPN interruption
- Manual stop
- Network failure
9. Important Prometheus Metrics
| Metric |
Query |
Purpose |
| Throughput |
sum(increase(http_server_requests_seconds_count[1m]))
|
Total Requests |
| Latency |
rate(http_server_requests_seconds_sum[5m])
|
Response Time |
| CPU |
process_cpu_usage
|
Application CPU Usage |
| Memory |
jvm_memory_used_bytes
|
Heap Usage |
| DB Pool Usage |
hikaricp_connections_active /
hikaricp_connections_max
|
Connection Pool Saturation |
10. Database Monitoring Queries
| Panel |
Query |
Purpose |
| DB Pool Usage |
sum(hikaricp_connections_active)
/ sum(hikaricp_connections_max)
|
Detect saturation |
| DB Pending Connections |
max(hikaricp_connections_pending)
|
Detect waiting threads |
| DB Request Rate |
rate(spring_data_repository_invocations_seconds_count[1m])
|
DB Throughput |
11. API Metrics
| Metric |
Query |
| Throughput |
sum(increase(http_server_requests_seconds_count[1m]))
|
| Endpoint Throughput |
sum by(uri)
(rate(http_server_requests_seconds_count[1m]))
|
| Traffic Per Pod |
sum(rate(http_server_requests_seconds_count[1m]))
by (pod)
|
12. Latency Metrics
Average Latency by Endpoint
sum by(uri)
(rate(http_server_requests_seconds_sum[5m]))
/
sum by(uri)
(rate(http_server_requests_seconds_count[5m]))
P95 Latency
histogram_quantile(
0.95,
rate(http_server_requests_seconds_bucket[5m])
)
Interview Tip:
Production systems rely more on P95 and P99 latency than average latency.
13. Prometheus Interview Questions
| Question |
Answer |
| What is Prometheus? |
Time-series monitoring system using pull model. |
| Pull vs Push? |
Prometheus scrapes metrics. PushGateway is used for batch jobs. |
| What is a Time Series? |
Metric + Labels + Timestamp. |
| rate() vs increase()? |
rate = per second average, increase = total count. |
| What is histogram_quantile? |
Calculates P95/P99 latency. |
14. Grafana Interview Questions
| Question |
Answer |
| What is Grafana? |
Visualization platform for monitoring data. |
| What is a Panel? |
Visualization unit such as Graph, Table, Gauge. |
| How do you troubleshoot production issues? |
Check latency, throughput, errors, CPU, memory. |
15. PushGateway Interview Questions
| Question |
Answer |
| What is PushGateway? |
Allows short-lived jobs to push metrics. |
| When should it be used? |
Batch Jobs, Cron Jobs, JMeter Tests. |
| Why not use for APIs? |
Metrics become stale and lifecycle isn't managed automatically. |
16. Scenario Based Questions
Scenario 1: Latency High but CPU Normal
Possible Causes:
- Database latency
- Network latency
- Thread pool exhaustion
- External API slowness
Conclusion:
Latency without CPU increase usually indicates I/O bottleneck.
Scenario 2: Throughput Drops Suddenly
Possible Causes:
- Pod crash
- Load balancer issue
- Network blackhole
- Routing issue
Query:
count(kube_pod_status_phase{phase="Running"})
Scenario 3: Error Rate Spike During Chaos
rate(
http_server_requests_seconds_count
{status=~"5.."}[1m]
)
Check:
- Retry metrics
- DB pool saturation
- Network issues
- Pod health
Scenario 4: DB Connection Pool Exhaustion
hikaricp_connections_active /
hikaricp_connections_max
If close to 1:
- Increase pool size
- Optimize queries
- Add caching
Scenario 5: Memory Continuously Increasing
jvm_memory_used_bytes{area="heap"}
Possible Causes:
- Memory Leak
- Cache Growth
- Large Object Retention
Scenario 6: High Retry Count but No Errors
Retries are masking transient failures.
Possible causes:
- DNS Issues
- Network Glitches
- Temporary DB Latency
Scenario 7: High Cardinality Issue
Bad:
uri="/user/12345"
Good:
uri="/user/{id}"
17. Chaos Engineering Impact Analysis
| Phase |
Observation |
| Before |
Stable Metrics |
| During |
Latency ? Errors ? |
| After |
Automatic Recovery |
Senior Interview Answer:
We used Prometheus for metrics collection, Grafana for visualization,
PushGateway for batch workloads, and Gremlin for chaos testing.
We monitored latency, throughput, JVM metrics, CPU, memory, retries,
database connection pools, and pod health. During chaos experiments,
we correlated Grafana dashboards with JMeter load and Gremlin fault
injection to validate application resilience and recovery behavior.