Prometheus, Grafana, PushGateway & Chaos Engineering

1. PushGateway Configuration

management:
  metrics:
    export:
      pushgateway:
        enabled: true
        base-url: http://pushgateway:9091
        job: ref-legacy-api-container
        push-rate: 10s
        shutdown-operation: DELETE
        grouping-key:
          instance: ${HOSTNAME}

What This Configuration Means

Best Practice: Always configure shutdown-operation: DELETE to avoid stale metrics.

2. PushGateway Deployment

Docker Setup

docker run -d \
-p 9091:9091 \
--name pushgateway \
prom/pushgateway

Access:

http://localhost:9091

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment

metadata:
  name: pushgateway

spec:
  replicas: 1

  selector:
    matchLabels:
      app: pushgateway

  template:
    metadata:
      labels:
        app: pushgateway

    spec:
      containers:
      - name: pushgateway
        image: prom/pushgateway

        ports:
        - containerPort: 9091

3. Prometheus Configuration

scrape_configs:

- job_name: 'pushgateway'

  honor_labels: true

  static_configs:
  - targets:
      - pushgateway:9091

Why honor_labels: true?

Prevents Prometheus from overwriting labels pushed by PushGateway.

4. Monitoring Architecture

Spring Boot Application
          |
          | Push Metrics
          V
      PushGateway
          |
          | Scrape
          V
      Prometheus
          |
          V
       Grafana

5. Chaos Engineering Architecture

JMeter (Load Testing)
        |
        V
Spring Boot Application
        |
        V
Kubernetes Cluster
        |
        V
Gremlin Chaos Experiments
        |
        V
Prometheus
        |
        V
Grafana Dashboard

6. Grafana Dashboard Panels

Panel Purpose Interpretation
Latency Response Time Spike indicates performance degradation
Throughput Requests per second Drop indicates load handling issue
Error Rate Failed Requests Increase indicates failures
CPU Resource Consumption High CPU means overload
Memory Heap Usage Continuous increase may indicate leak
Pod Health Restart Count Frequent restarts indicate instability

7. RCA Timeline Example

Time Event
4:52 PM Gremlin Blackhole Started
4:55 PM Latency Increased
4:58 PM Chaos Interrupted
4:59 PM System Recovery Observed

8. Root Cause Analysis

Primary Cause:

9. Important Prometheus Metrics

Metric Query Purpose
Throughput sum(increase(http_server_requests_seconds_count[1m])) Total Requests
Latency rate(http_server_requests_seconds_sum[5m]) Response Time
CPU process_cpu_usage Application CPU Usage
Memory jvm_memory_used_bytes Heap Usage
DB Pool Usage hikaricp_connections_active / hikaricp_connections_max Connection Pool Saturation

10. Database Monitoring Queries

Panel Query Purpose
DB Pool Usage sum(hikaricp_connections_active) / sum(hikaricp_connections_max) Detect saturation
DB Pending Connections max(hikaricp_connections_pending) Detect waiting threads
DB Request Rate rate(spring_data_repository_invocations_seconds_count[1m]) DB Throughput

11. API Metrics

Metric Query
Throughput sum(increase(http_server_requests_seconds_count[1m]))
Endpoint Throughput sum by(uri) (rate(http_server_requests_seconds_count[1m]))
Traffic Per Pod sum(rate(http_server_requests_seconds_count[1m])) by (pod)

12. Latency Metrics

Average Latency by Endpoint

sum by(uri)
(rate(http_server_requests_seconds_sum[5m]))
/
sum by(uri)
(rate(http_server_requests_seconds_count[5m]))

P95 Latency

histogram_quantile(
0.95,
rate(http_server_requests_seconds_bucket[5m])
)
Interview Tip: Production systems rely more on P95 and P99 latency than average latency.

13. Prometheus Interview Questions

Question Answer
What is Prometheus? Time-series monitoring system using pull model.
Pull vs Push? Prometheus scrapes metrics. PushGateway is used for batch jobs.
What is a Time Series? Metric + Labels + Timestamp.
rate() vs increase()? rate = per second average, increase = total count.
What is histogram_quantile? Calculates P95/P99 latency.

14. Grafana Interview Questions

Question Answer
What is Grafana? Visualization platform for monitoring data.
What is a Panel? Visualization unit such as Graph, Table, Gauge.
How do you troubleshoot production issues? Check latency, throughput, errors, CPU, memory.

15. PushGateway Interview Questions

Question Answer
What is PushGateway? Allows short-lived jobs to push metrics.
When should it be used? Batch Jobs, Cron Jobs, JMeter Tests.
Why not use for APIs? Metrics become stale and lifecycle isn't managed automatically.

16. Scenario Based Questions

Scenario 1: Latency High but CPU Normal

Possible Causes: Conclusion: Latency without CPU increase usually indicates I/O bottleneck.

Scenario 2: Throughput Drops Suddenly

Possible Causes: Query:
count(kube_pod_status_phase{phase="Running"})

Scenario 3: Error Rate Spike During Chaos

rate(
http_server_requests_seconds_count
{status=~"5.."}[1m]
)
Check:

Scenario 4: DB Connection Pool Exhaustion

hikaricp_connections_active /
hikaricp_connections_max
If close to 1:

Scenario 5: Memory Continuously Increasing

jvm_memory_used_bytes{area="heap"}
Possible Causes:

Scenario 6: High Retry Count but No Errors

Retries are masking transient failures. Possible causes:

Scenario 7: High Cardinality Issue

Bad:
uri="/user/12345"
Good:
uri="/user/{id}"

17. Chaos Engineering Impact Analysis

Phase Observation
Before Stable Metrics
During Latency ? Errors ?
After Automatic Recovery
Senior Interview Answer: We used Prometheus for metrics collection, Grafana for visualization, PushGateway for batch workloads, and Gremlin for chaos testing. We monitored latency, throughput, JVM metrics, CPU, memory, retries, database connection pools, and pod health. During chaos experiments, we correlated Grafana dashboards with JMeter load and Gremlin fault injection to validate application resilience and recovery behavior.