OpenSearch / Elasticsearch

Connect OpenSearch or Elasticsearch so OpenSRE can search application logs and analytics indices during investigations

Overview

OpenSRE queries OpenSearch (or Elasticsearch) to retrieve application logs, error events, and analytics records — pulling concrete log lines into the investigation alongside metrics and traces from your other observability tools.

For Amazon OpenSearch Service (AWS ES) domain specifics, see also AWS Elasticsearch.

Prerequisites

• OpenSearch 1.x/2.x or Elasticsearch 7.x/8.x reachable from the machine running OpenSRE
• The cluster URL (e.g. https://my-cluster.us-east-1.es.amazonaws.com)
• Credentials for one of:
  • HTTP Basic Auth (username and password) — typical for self-hosted OpenSearch
  • API key — typical for Elastic Cloud
  • No auth — only when the cluster has the security plugin disabled

OpenSearch authenticates clients via Basic Auth by default; the security plugin does not natively issue API keys (opensearch-project/security#4009). Most self-hosted clusters use Basic Auth.

Setup

Option 1: Interactive CLI / onboard

opensre integrations setup opensearch
# or
opensre onboard

Pick OpenSearch / Elasticsearch. Choose auth mode: Username + Password, API key, or None.

Option 2: Environment variables

OPENSEARCH_URL=https://my-cluster.example.com

# Basic Auth (typical for self-hosted OpenSearch)
OPENSEARCH_USERNAME=admin
OPENSEARCH_PASSWORD=secret

# API key (typical for Elastic Cloud — use instead of username/password)
OPENSEARCH_API_KEY=your-api-key

# Optional
OPENSEARCH_INDEX_PATTERN=*
OPENSEARCH_MAX_RESULTS=

Variable	Default	Description
OPENSEARCH_URL	—	Required. Base URL of your cluster
OPENSEARCH_USERNAME	—	Basic auth username
OPENSEARCH_PASSWORD	—	Basic auth password
OPENSEARCH_API_KEY	—	API key (Authorization: ApiKey …)
OPENSEARCH_INDEX_PATTERN	*	Default index pattern
OPENSEARCH_MAX_RESULTS	—	Optional result cap

Use API key or basic auth, not both — verify rejects both set together. If only one of username/password is set, no Authorization header is emitted.

Option 3: Persistent store

{
  "version": 1,
  "integrations": [
    {
      "id": "opensearch-prod",
      "service": "opensearch",
      "status": "active",
      "credentials": {
        "url": "https://my-cluster.example.com",
        "username": "admin",
        "password": "secret"
      }
    }
  ]
}

Credentials

Prefer a dedicated read-only user (or API key) scoped to the indices OpenSRE should search. For AWS ES, use fine-grained access control with an internal user — see AWS Elasticsearch.

Quick local test with Docker

docker run -d --name opensearch-dev -p 127.0.0.1:9201:9200 \
  -e "discovery.type=single-node" \
  -e "DISABLE_SECURITY_PLUGIN=true" \
  -e "OPENSEARCH_JAVA_OPTS=-Xms256m -Xmx256m" \
  opensearchproject/opensearch:2

i=0
until curl -sf -o /dev/null http://localhost:9201; do
  i=$((i + 1))
  [ "$i" -ge 30 ] && { echo "ERROR: OpenSearch never became ready" >&2; exit 1; }
  sleep 2
done

Seed the index with an error and a non-error log line — both are needed to match the two-record output shown below. The tools sort on @timestamp, so use that field name, not timestamp:

curl -sf -X POST "http://localhost:9201/logs-app/_doc" -H "Content-Type: application/json" -d '{
  "@timestamp": "'"$(date -u +%Y-%m-%dT%H:%M:%S)"'Z",
  "level": "ERROR",
  "message": "payment-service: connection timeout while calling billing-api",
  "service": "payment-service"
}' > /dev/null &&
curl -sf -X POST "http://localhost:9201/logs-app/_doc" -H "Content-Type: application/json" -d '{
  "@timestamp": "'"$(date -u +%Y-%m-%dT%H:%M:%S)"'Z",
  "level": "INFO",
  "message": "payment-service: request completed",
  "service": "payment-service"
}' > /dev/null &&
curl -sf -X POST "http://localhost:9201/logs-app/_refresh" > /dev/null || { echo "ERROR: seeding failed" >&2; exit 1; }

export OPENSEARCH_URL="http://localhost:9201"

Verify:

opensre integrations verify opensearch

SERVICE    │ SOURCE    │ STATUS   │ DETAIL
opensearch │ local env │ ✓ passed │ Configured for OpenSearch at
           │           │          │ http://localhost:9201.

Register the local instance through the supported setup flow:

opensre integrations setup opensearch

Trigger a real investigation against the seeded index — this is the actual supported entrypoint, not an internal import:

opensre investigate --input-json '{
  "alert_name": "PaymentServiceErrors",
  "severity": "critical",
  "commonAnnotations": {
    "summary": "payment-service throwing connection timeout errors calling billing-api"
  }
}'

Real output from a run against this exact seeded index (edited for length):

Loading integrations: opensearch
↳ Elasticsearch · query elasticsearch logs (service:payment-service AND (timeout OR "connection" OR error)) → total: 1
↳ Elasticsearch · query elasticsearch logs (service:billing-api AND (error OR exception OR timeout ...)) → total: 0

root_cause: "billing-api appears to be non-functional or absent -- it has
produced zero logs across the entire 6-hour observation window while its
dependent caller (payment-service) has begun reporting connection
timeouts against it. The most likely explanation is that billing-api
pods are not running, not ready, or otherwise not serving traffic..."

The agent found the seeded error via query_elasticsearch_logs and didn’t need query_opensearch_analytics for this alert — both tools share the same credentials and index, so either is reachable from the same setup.

Teardown:

docker rm -f opensearch-dev

Investigation tools

Tool	What it does
query_opensearch_analytics	Search analytics / log indices
query_elasticsearch_logs	Log lines matching a query in a time range, plus error-like lines

Both become available once the opensearch integration is configured.

Verify

opensre integrations verify opensearch

CLI verify is a configuration-coherence check (URL + XOR auth). Onboard may also call GET /_cluster/health for live feedback.

Troubleshooting

Symptom	Fix
Both API key and basic auth set	Clear one method — verify rejects both
403 Forbidden on AWS ES	Enable fine-grained access control; use internal user/password (not IAM SigV4)
No tools in plan	Confirm verify passes and sources include opensearch
Empty results	Check OPENSEARCH_INDEX_PATTERN and index permissions

Security

• Prefer least-privilege read users / API keys.
• Do not use the master user for OpenSRE.
• Store credentials out of source control.