Splunk Interview Notes for Spring Boot Microservices

1. What is Splunk?

Splunk is a centralized log aggregation, monitoring, searching, and analytics platform used to collect machine-generated data from applications, servers, containers, cloud services, and infrastructure.

Simple Definition:
Splunk = Google for your logs

2. Why Splunk is Used in Production?

3. Splunk Architecture

Spring Boot App
       |
       v
Application Log File
(/var/log/app.log)
       |
       v
Universal Forwarder
       |
       v
Indexer
       |
       v
Search Head
       |
       v
Dashboard / Alerts

4. Key Splunk Components

Component Purpose
Forwarder Collects logs from servers
Indexer Indexes and stores logs
Search Head Executes searches and dashboards
Deployment Server Manages forwarders
Cluster Master Manages indexer clusters

5. Types of Logs in Spring Boot

INFO  Order Created
ERROR Payment Failed
WARN  Retry Attempt
DEBUG API Response

6. Logging Frameworks

Framework Description
Logback Default Spring Boot Logger
Log4j2 High-performance logging
SLF4J Logging abstraction layer

7. Structured Logging (Best Practice)

Always use JSON logging.

<encoder class=
"net.logstash.logback.encoder.LogstashEncoder"/>
Sample Output:
{
 "timestamp":"2026-04-16T10:00:00",
 "service":"order-service",
 "level":"ERROR",
 "traceId":"abc123",
 "message":"Payment failed"
}

8. Correlation ID / Trace ID

Most important concept in Microservices logging.

MDC.put(
 "traceId",
 UUID.randomUUID().toString()
);
Log Pattern:
%X{traceId}
Benefits:

9. Splunk Search Queries (SPL)

Search Errors:
index=app_logs level=ERROR
Search Service Logs:
index=app_logs service=payment-service
Search By TraceId:
traceId=abc123
Last 15 Minutes:
index=app_logs earliest=-15m
Search Exceptions:
"NullPointerException"
OR
"TimeoutException"
Count Errors:
index=app_logs level=ERROR
| stats count by service
Top 10 Errors:
index=app_logs level=ERROR
| top limit=10 exception

10. Spring Boot Logback Configuration

<appender name="FILE"
class=
"ch.qos.logback.core.rolling.RollingFileAppender">

<file>
/var/log/app.log
</file>

</appender>

11. Sending Logs to Splunk

Option 1 - Universal Forwarder

./splunk add monitor
/var/log/app.log

Option 2 - HTTP Event Collector (HEC)

restTemplate.postForObject(
"https://splunk:8088/services/collector",
payload,
String.class
);

Option 3 - Logstash

Spring Boot
   |
Logstash
   |
Splunk

12. Real Production RCA Example

Issue: Payment failures reported. Steps:
service=payment-service
level=ERROR
Find TraceId:
traceId=xyz789
Track Across Services:
Order Service
      |
Payment Service
      |
Notification Service
Root Cause Found:
Database Timeout
Resolution:
Increase DB Connection Pool
Optimize Query

13. Splunk Dashboards

Common Dashboards:

14. Splunk Alerts

Example:
index=app_logs level=ERROR
| stats count
Alert Condition:
count > 50
within 5 minutes
Notification:

15. Splunk vs ELK vs Prometheus

Tool Purpose
Splunk Log Analytics
ELK Open Source Log Analytics
Prometheus Metrics Collection
Grafana Visualization

16. Interview Questions

17. Quick Interview One-Liners

Question One-Line Answer
Splunk Centralized log monitoring and analytics platform.
Indexer Stores and indexes logs.
Search Head Provides search and dashboard functionality.
Forwarder Collects logs from servers.
TraceId Unique identifier for tracking requests across services.
Structured Logging Logging in JSON format for easy parsing.
HEC HTTP Event Collector for direct log ingestion.
RCA Root Cause Analysis of production issues.
SPL Search Processing Language used in Splunk.
Dashboard Visual representation of log data.
Alert Notification triggered by search conditions.
Best Practice Always use JSON logs with traceId.
Splunk vs Prometheus Logs vs Metrics.
Splunk vs ELK Commercial vs Open Source logging solution.

18. 5-Minute Interview Answer

Splunk is a centralized log aggregation, monitoring, and analytics platform used to collect, index, search, and visualize logs from distributed systems. In Spring Boot applications, logs are generated using Logback or Log4j2 and are typically written in structured JSON format. These logs are forwarded to Splunk through a Universal Forwarder, HTTP Event Collector (HEC), or Logstash pipeline. We include traceId or correlationId in logs to track requests across microservices. Splunk allows real-time log searching using SPL queries, dashboard creation, alerting, and root cause analysis. It plays a critical role in monitoring production systems, troubleshooting failures, and maintaining application reliability.