Splunk with Spring Boot - Complete Reference Guide

What is Splunk?

Splunk is a centralized log monitoring and analytics platform used to collect, index, search, analyze, and visualize machine-generated data from applications, servers, containers, cloud platforms, and infrastructure.

Splunk = "Google for your logs"

Why Splunk is Used in Production

Types of Logs in Spring Boot

Log Type Description
Application Logs General application execution logs.
Exception Logs Runtime errors and exceptions.
Access Logs HTTP request and response details.
Business Logs Order creation, payment processing, notifications.

Logging Frameworks

Splunk Search Queries

1. Search Error Logs

index=app_logs service=order-service level=ERROR

Filters

Search by Correlation ID (Most Important)

In Microservices Architecture, every request should carry a Correlation ID / Trace ID.

traceId=abc123
Helps trace a request across multiple services.

Search Exception Logs

"NullPointerException" OR "TimeoutException"

Time-Based Analysis

index=app_logs earliest=-15m

Returns logs generated in the last 15 minutes.

Count Errors by Service

index=app_logs level=ERROR | stats count by service

Spring Boot Logging Architecture

Spring Boot Application
          |
          v
      Log File
          |
          v
Splunk Universal Forwarder
          |
          v
     Splunk Indexer
          |
          v
      Splunk UI

Key Components

Component Purpose
Spring Boot App Generates logs.
Log File Stores application logs.
Universal Forwarder Sends logs to Splunk.
Indexer Indexes and stores logs.
Dashboard Visualization and monitoring.

Structured Logging (Recommended)

JSON logging is the preferred approach for Splunk integration.

Logback Configuration

<encoder class=
"net.logstash.logback.encoder.LogstashEncoder"/>

Sample JSON Log

{
  "timestamp":"2026-04-16T10:00:00",
  "level":"ERROR",
  "service":"order-service",
  "traceId":"abc123",
  "message":"Payment failed"
}

Add Correlation ID (TraceId)

Generate TraceId

MDC.put(
   "traceId",
   UUID.randomUUID().toString()
);

Logback Pattern

%X{traceId}

Configure Log Output

<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">

    <file>/var/log/app.log</file>

</appender>

Install Splunk Universal Forwarder

The Splunk Forwarder monitors log files and sends logs to Splunk Indexers.

./splunk add monitor /var/log/app.log

Methods to Send Logs to Splunk

1. HTTP Event Collector (HEC)

RestTemplate.postForObject(
"https://splunk:8088/services/collector",
log,
String.class
);

2. Fluentd / Logstash

3. Splunk Cloud

Production RCA Example

Issue

Payment failures in Production

Step 1

service=payment-service level=ERROR

Step 2

traceId=xyz789

Step 3

Track flow across services:

Step 4

Identify Root Cause:

Best Practices

Splunk vs Prometheus vs Grafana

Tool Primary Purpose
Splunk Log Analytics
Prometheus Metrics Collection
Grafana Visualization & Dashboards

5-Minute Interview Answer

Splunk is a centralized log monitoring and analytics platform used to collect, search, analyze, and visualize logs from distributed applications and infrastructure. In Spring Boot applications, logs are generated using frameworks like Logback and are usually formatted as structured JSON logs.

These logs are forwarded to Splunk through Universal Forwarders, Logstash, or HTTP Event Collector (HEC). Correlation IDs such as traceId are included to trace requests across microservices.

Splunk enables developers and support teams to perform real-time monitoring, troubleshooting, root cause analysis (RCA), dashboard creation, and alerting, making it an essential tool in production environments.

Interview Questions

  1. What is Splunk and why is it used?
  2. What is a Splunk Index?
  3. What is a Universal Forwarder?
  4. What is HTTP Event Collector (HEC)?
  5. Why is TraceId important?
  6. How do you integrate Spring Boot with Splunk?
  7. Difference between Splunk and ELK?
  8. Difference between Splunk and Prometheus?
  9. How do you perform RCA using Splunk?
  10. What are best practices for logging in Microservices?