> ## Documentation Index
> Fetch the complete documentation index at: https://opensre.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Splunk

> Connect Splunk so OpenSRE can search logs using SPL during investigations

OpenSRE queries Splunk using the REST API to surface relevant log evidence during alert investigations — searching indexes with SPL, correlating error patterns with incidents, and identifying root causes.

## Prerequisites

* Splunk Enterprise or Splunk Cloud instance (version 8.x or later)
* REST API access on port 8089
* A bearer token with search capability (see [Generating a Bearer Token](#generating-a-bearer-token))

## Setup

### Option 1: Interactive CLI

```bash theme={null}
opensre integrations setup
```

Select **Splunk** when prompted and provide your REST API base URL and bearer token.

### Option 2: Environment variables

Add to your `.env`:

```bash theme={null}
SPLUNK_URL=https://splunk.corp.com:8089    # REST API base URL (port 8089 default)
SPLUNK_TOKEN=your-bearer-token             # API bearer token (NOT an HEC token)
SPLUNK_INDEX=main                          # Default index to search (optional)
SPLUNK_VERIFY_SSL=true                     # Set false to skip SSL verification (optional)
SPLUNK_CA_BUNDLE=/etc/ssl/certs/corp-ca.pem  # Path to custom CA bundle (optional)
```

| Variable            | Default | Description                                                     |
| ------------------- | ------- | --------------------------------------------------------------- |
| `SPLUNK_URL`        | —       | **Required.** REST API base URL including port                  |
| `SPLUNK_TOKEN`      | —       | **Required.** Bearer token with search capability               |
| `SPLUNK_INDEX`      | `main`  | Default index searched when no index is specified in the alert  |
| `SPLUNK_VERIFY_SSL` | `true`  | Set to `false` to disable SSL verification (dev/local only)     |
| `SPLUNK_CA_BUNDLE`  | —       | Path to a PEM CA bundle for enterprise self-signed certificates |

### Option 3: Persistent store

```json theme={null}
{
  "version": 1,
  "integrations": [
    {
      "id": "splunk-prod",
      "service": "splunk",
      "status": "active",
      "credentials": {
        "base_url": "https://splunk.corp.com:8089",
        "token": "your-bearer-token",
        "index": "main",
        "verify_ssl": true,
        "ca_bundle": "/etc/ssl/certs/corp-ca.pem"
      }
    }
  ]
}
```

### Multi-instance setup

To connect multiple Splunk instances (e.g. separate prod and staging clusters):

```bash theme={null}
SPLUNK_INSTANCES='[
  {"name":"prod","tags":{"env":"prod"},"credentials":{"base_url":"https://splunk-prod:8089","token":"prod-token","index":"prod"}},
  {"name":"staging","tags":{"env":"staging"},"credentials":{"base_url":"https://splunk-staging:8089","token":"staging-token","index":"staging"}}
]'
```

When `SPLUNK_INSTANCES` is set it overrides the single-instance `SPLUNK_URL` / `SPLUNK_TOKEN` variables.

## Generating a bearer token

OpenSRE uses bearer tokens — not basic auth and not HEC tokens. To generate one:

**Via the Splunk UI:**

1. Go to **Settings** → **Tokens**
2. Click **New Token**
3. Set a name (e.g. `opensre`) and an expiry date
4. Copy the generated token

**Via the REST API** (replace `<PASSWORD>` with your admin password):

```bash theme={null}
curl -sk -u admin:<PASSWORD> \
  https://splunk.corp.com:8089/services/authorization/tokens \
  -X POST \
  --data-urlencode "name=opensre" \
  --data-urlencode "expires_on=+90d" \
  --data-urlencode "output_mode=json" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['entry'][0]['content']['token'])"
```

The token needs the `search` capability. The `admin` role includes this by default. For a dedicated service account, ensure the role includes:

* `search`
* `read_splunkd_private_settings` (needed for the verify call against `/services/server/info`)

## Verify

```bash theme={null}
opensre integrations verify splunk
```

Expected output:

```
Service: splunk
Status: passed
Detail: Connected to Splunk 9.x.x
```

## How queries are generated

OpenSRE builds SPL queries deterministically from the alert payload — the LLM selects which tool to call but never writes the query itself. This keeps investigations reproducible and auditable.

Query construction priority:

| Priority | Source                                                     | Example                                               |
| -------- | ---------------------------------------------------------- | ----------------------------------------------------- |
| 1        | `annotations.splunk_query` — verbatim SPL from your alert  | `index=prod "PaymentTimeout" \| head 50`              |
| 2        | `annotations.query` or `annotations.log_query`             | Any pre-populated query field                         |
| 3        | `error_message` field — keyword search built automatically | `search index=main "NullPointerException" \| head 50` |
| 4        | `alert_name` — last-resort keyword search                  | `search index=main "payments-error-spike" \| head 50` |
| 5        | Fallback — index scan                                      | `search index=main \| head 50`                        |

To pass a specific SPL query through an alert, set `commonAnnotations.splunk_query`:

```json theme={null}
{
  "alert_name": "Payment service errors",
  "commonAnnotations": {
    "splunk_query": "index=prod sourcetype=app_logs \"NullPointerException\" | head 50"
  }
}
```

## Test with the built-in alert template

```bash theme={null}
opensre investigate --template splunk
```

This runs a synthetic investigation using a pre-built alert fixture — no live alert infrastructure needed.

## Troubleshooting

| Symptom                             | Fix                                                                                             |
| ----------------------------------- | ----------------------------------------------------------------------------------------------- |
| `SSL: CERTIFICATE_VERIFY_FAILED`    | Set `SPLUNK_CA_BUNDLE=/path/to/corp-ca.pem` (preferred) or `SPLUNK_VERIFY_SSL=false` (dev only) |
| `HTTP 401 Unauthorized`             | Token expired or was generated with the wrong account — regenerate                              |
| `HTTP 403 Forbidden`                | Token lacks `search` capability — check the role assigned to the token                          |
| Empty search results                | Data may not have been ingested yet, or the index name is wrong                                 |
| `Connection refused` on port 8089   | Splunk management port may be firewalled; confirm network access                                |
| `opensre integrations verify` fails | Check `SPLUNK_URL` includes the protocol and port (`https://host:8089`)                         |

## Security best practices

* Use a **read-only bearer token** — never use an admin token in production.
* Store `SPLUNK_TOKEN` in `.env` or the credential store, not in source code or CI logs.
* Prefer a dedicated `opensre` service account with only the `search` capability.
* For enterprise self-signed certificates, set `SPLUNK_CA_BUNDLE` to the CA bundle path rather than disabling verification entirely.
* Set `SPLUNK_VERIFY_SSL=false` only in local or dev environments when you cannot supply a CA bundle.
* Rotate tokens on a schedule and revoke them when no longer needed.