Splunk

Connect Splunk so OpenSRE can search logs using SPL during investigations

Overview

OpenSRE queries Splunk over the REST API during investigations. The agent calls query_splunk_logs with a planner-supplied SPL query.

There is no opensre integrations setup splunk handler today. Configure Splunk via the onboarding wizard, environment variables, or the integration store, then verify with opensre integrations verify splunk.

Prerequisites

• Splunk Enterprise or Splunk Cloud (version 8.x or later)
• REST API access on port 8089
• A bearer token with search capability (see Credentials)

Setup

Option 1: Onboarding wizard

opensre onboard

Select Splunk under Observability and provide the REST API base URL and bearer token. The wizard validates and writes the integration store (token stays in the credential store; URL/index/SSL settings sync to .env).

Option 2: Environment variables

SPLUNK_URL=https://splunk.corp.com:8089    # REST API base URL (port 8089 default)
SPLUNK_TOKEN=your-bearer-token             # API bearer token (NOT an HEC token)
SPLUNK_INDEX=main                          # Default index to search (optional)
SPLUNK_VERIFY_SSL=true                     # Set false to skip SSL verification (optional)
SPLUNK_CA_BUNDLE=/etc/ssl/certs/corp-ca.pem  # Path to custom CA bundle (optional)

Variable	Default	Description
SPLUNK_URL	—	Required. REST API base URL including port
SPLUNK_TOKEN	—	Required. Bearer token with search capability
SPLUNK_INDEX	main	Default index when the tool does not override index
SPLUNK_VERIFY_SSL	true	Set to false to disable SSL verification (dev/local only)
SPLUNK_CA_BUNDLE	—	Path to a PEM CA bundle for enterprise self-signed certificates

Option 3: Persistent store

{
  "version": 1,
  "integrations": [
    {
      "id": "splunk-prod",
      "service": "splunk",
      "status": "active",
      "credentials": {
        "base_url": "https://splunk.corp.com:8089",
        "token": "your-bearer-token",
        "index": "main",
        "verify_ssl": true,
        "ca_bundle": "/etc/ssl/certs/corp-ca.pem"
      }
    }
  ]
}

Option 4: Multi-instance

SPLUNK_INSTANCES='[
  {"name":"prod","tags":{"env":"prod"},"credentials":{"base_url":"https://splunk-prod:8089","token":"prod-token","index":"prod"}},
  {"name":"staging","tags":{"env":"staging"},"credentials":{"base_url":"https://splunk-staging:8089","token":"staging-token","index":"staging"}}
]'

When SPLUNK_INSTANCES is set it overrides the single-instance SPLUNK_URL / SPLUNK_TOKEN variables.

Credentials

OpenSRE uses bearer tokens — not basic auth and not HEC tokens.

Via the Splunk UI:

1. Go to Settings → Tokens
2. Click New Token
3. In User, pick the existing Splunk user the token authenticates as
4. Fill in Audience (required) and an expiry date
5. Copy the generated token

Via the REST API (replace <PASSWORD> with your admin password and <SPLUNK_USER> with the service account the token should authenticate as):

curl -sk -u admin:<PASSWORD> \
  https://splunk.corp.com:8089/services/authorization/tokens \
  -X POST \
  --data-urlencode "name=<SPLUNK_USER>" \
  --data-urlencode "audience=OpenSRE" \
  --data-urlencode "expires_on=+90d" \
  --data-urlencode "output_mode=json" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['entry'][0]['content']['token'])"

name is the existing Splunk user the token authenticates as, not an arbitrary label — Splunk rejects an unrecognized name with User "..." does not exist. The token inherits that user’s roles, so point it at a least-privilege service account rather than admin. audience is also required (a short free-text description of the token’s purpose); omitting it fails with The following required arguments are missing: audience.

The token needs the search capability. The admin role includes this by default. For a dedicated service account, ensure the role includes:

• search
• read_splunkd_private_settings (needed for the verify call against /services/server/info)

Quick local test with Docker

docker run -d --name splunk-dev --platform linux/amd64 \
  -p 127.0.0.1:8000:8000 \
  -p 127.0.0.1:8089:8089 \
  -e SPLUNK_START_ARGS=--accept-license \
  -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \
  -e SPLUNK_PASSWORD=VerifyPass123! \
  splunk/splunk:latest

i=0
until docker logs splunk-dev 2>&1 | grep -q "Ansible playbook complete"; do
  i=$((i + 1))
  if [ "$i" -ge 180 ]; then
    echo "Splunk never became ready; check: docker logs splunk-dev" >&2
    break
  fi
  sleep 5
done

splunk/splunk publishes amd64 only, which is why --platform linux/amd64 is already on the command above. On Apple Silicon it runs under emulation and can take 15 minutes or more to report ready; on native amd64 the flag is a harmless no-op. Without SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com alongside SPLUNK_START_ARGS=--accept-license, the container exits immediately with License not accepted — newer images require both, not just the license flag documented in older Splunk Docker guides.

Seed a test event and create a token:

curl -sk -u admin:VerifyPass123! \
  "https://localhost:8089/services/receivers/simple?index=main&sourcetype=opensre_verify" \
  --data-binary "$(date -u '+%Y-%m-%d %H:%M:%S') ERROR payment-service: connection refused to billing-api during checkout"

SPLUNK_TOKEN=$(curl -sk -u admin:VerifyPass123! \
  https://localhost:8089/services/authorization/tokens \
  -X POST \
  --data-urlencode "name=admin" \
  --data-urlencode "audience=OpenSRE verification" \
  --data-urlencode "expires_on=+90d" \
  --data-urlencode "output_mode=json" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['entry'][0]['content']['token'])")

export SPLUNK_URL="https://localhost:8089"
export SPLUNK_TOKEN
export SPLUNK_VERIFY_SSL=false
export SPLUNK_INDEX=main
unset SPLUNK_INSTANCES SPLUNK_CA_BUNDLE

Then point OpenSRE at an empty store, so the container is the only Splunk either command can reach:

export OPENSRE_DEMO_STORE_DIR="$(mktemp -d /tmp/opensre-splunk-demo.XXXXXX)"
export OPENSRE_INTEGRATIONS_STORE_PATH="$OPENSRE_DEMO_STORE_DIR/integrations.json"

Do this before verifying, and keep it for the rest of the recipe. A saved Splunk record wins over SPLUNK_URL / SPLUNK_TOKEN for opensre integrations verify, so with your real store in play the check can pass against production while appearing to test localhost — the SOURCE column reads local store instead of local env. opensre investigate fails the other way: it only falls through to env vars when the store has no records at all, so a single record for any unrelated service blocks env resolution entirely. An empty store is the one state both commands agree on. SPLUNK_INSTANCES overrides SPLUNK_URL / SPLUNK_TOKEN outright, which is why the block above clears it.

Verify:

opensre integrations verify splunk

SERVICE │ SOURCE    │ STATUS   │ DETAIL
splunk  │ local env │ ✓ passed │ Connected to Splunk 10.4.2

Trigger a real investigation against the seeded event. opensre investigate --print-template splunk prints this payload shape if you want to start from it:

opensre investigate --input-json '{
  "alert_source": "splunk",
  "alert_name": "PaymentServiceErrors",
  "severity": "critical",
  "commonAnnotations": {
    "summary": "payment-service errors detected",
    "splunk_query": "index=main sourcetype=opensre_verify | head 50"
  }
}'

Real output from a run against this exact local instance (edited for length). Other integrations still resolved from your own environment ride along harmlessly:

Loading integrations: telegram, splunk
↳ Splunk · query splunk logs

Triage complete: payment-service checkout failed with connection refused
to billing-api at 21:40:00 UTC; no billing-api logs are present in Splunk
during a 4-hour window across all indexes.

root_cause: payment-service is receiving TCP connection refused when
calling billing-api during checkout, and billing-api has emitted zero log
lines across all Splunk indexes for the last 4 hours. The most likely
explanation is that billing-api pods are not running or not listening,
though without cluster/deploy visibility the exact narrow cause cannot be
confirmed.

Teardown:

docker rm -f splunk-dev
rm -rf "$OPENSRE_DEMO_STORE_DIR"
unset OPENSRE_DEMO_STORE_DIR OPENSRE_INTEGRATIONS_STORE_PATH \
  SPLUNK_URL SPLUNK_TOKEN SPLUNK_VERIFY_SSL SPLUNK_INDEX

Investigation tools

query_splunk_logs

Argument	Required	Description
query	Yes	SPL search string (for example index=main error | head 50). Do not include a leading search keyword.
time_range_minutes	No	Look-back window in minutes (default 60)
limit	No	Max events to return (default 50)
index	No	Overrides the integration default index

The investigation planner writes the SPL query and calls the tool. OpenSRE does not build SPL from a fixed priority table.

When alert_source is splunk, OpenSRE auto-seeds the Splunk tool source before the ReAct loop. Seed calls use integration defaults if the planner has not supplied a query yet.

Verify

opensre integrations verify splunk

Expected output:

Service: splunk
Status: passed
Detail: Connected to Splunk 9.x.x

Troubleshooting

Symptom	Fix
SSL: CERTIFICATE_VERIFY_FAILED	Set SPLUNK_CA_BUNDLE=/path/to/corp-ca.pem (preferred) or SPLUNK_VERIFY_SSL=false (dev only)
HTTP 401 Unauthorized	Token expired or was generated with the wrong account — regenerate
HTTP 403 Forbidden	Token lacks search capability — check the role assigned to the token
Empty search results	Data may not have been ingested yet, or the index name is wrong
Connection refused on port 8089	Splunk management port may be firewalled; confirm network access
opensre integrations verify fails	Check SPLUNK_URL includes the protocol and port (https://host:8089)
opensre integrations setup splunk	Not available — use opensre onboard, env vars, or the store, then verify

Security

• Use a read-only bearer token — never use an admin token in production.
• Store SPLUNK_TOKEN in .env or the credential store, not in source code or CI logs.
• Prefer a dedicated opensre service account with only the search capability (plus read_splunkd_private_settings for verify).
• For enterprise self-signed certificates, set SPLUNK_CA_BUNDLE rather than disabling verification entirely.
• Set SPLUNK_VERIFY_SSL=false only in local or dev environments when you cannot supply a CA bundle.
• Rotate tokens on a schedule and revoke them when no longer needed.

Extras

Alert annotations as planner context

Putting a ready-made SPL string in the alert helps the planner. The built-in Splunk alert template includes commonAnnotations.splunk_query. The annotation is agent context (visible in the alert payload) — it is not executed by a separate deterministic builder.

{
  "alert_name": "Payment service errors",
  "alert_source": "splunk",
  "commonAnnotations": {
    "splunk_query": "index=prod sourcetype=app_logs \"NullPointerException\" | head 50"
  }
}

Sample alert template

There is no opensre investigate --template splunk. Use one of:

Action	Command
Print starter JSON (CLI)	opensre investigate --print-template splunk
Print starter JSON (REPL)	/template splunk
Run the sample alert (REPL)	/investigate splunk

Then edit the printed JSON and pass it with opensre investigate --input-json '…' or /investigate path/to/alert.json if you want a file-based run.