Alertmanager

Connect Alertmanager so OpenSRE can surface firing alerts and active silences during investigations

Overview

OpenSRE queries Alertmanager for firing, silenced, and inhibited alerts so investigations can correlate the triggering alert with other concurrent signals.

Prerequisites

• Alertmanager v0.20+ reachable from the machine running OpenSRE
• The Alertmanager URL (for example http://alertmanager.monitoring.svc:9093)
• Credentials if your instance uses authentication (bearer token or basic auth — not both)

Setup

Option 1: Interactive CLI

opensre integrations setup alertmanager

The wizard asks for:

1. Alertmanager URL — base URL of your Alertmanager instance
2. Authentication method — one of:
   • None — unauthenticated instances on an internal network
   • Bearer token — reverse proxy that accepts a token
   • Basic auth — username and password

Option 2: Environment variables

ALERTMANAGER_URL=http://alertmanager.monitoring.svc:9093

# Bearer token auth (optional)
ALERTMANAGER_BEARER_TOKEN=your-token

# Basic auth (optional)
ALERTMANAGER_USERNAME=admin
ALERTMANAGER_PASSWORD=secret

Variable	Default	Description
ALERTMANAGER_URL	—	Required. Base URL of your Alertmanager instance
ALERTMANAGER_BEARER_TOKEN	—	Bearer token for reverse-proxy auth
ALERTMANAGER_USERNAME	—	Basic auth username
ALERTMANAGER_PASSWORD	—	Basic auth password

Use at most one auth method. Setting both a bearer token and basic auth credentials is rejected.

Option 3: Persistent store

{
  "version": 1,
  "integrations": [
    {
      "id": "alertmanager-prod",
      "service": "alertmanager",
      "status": "active",
      "credentials": {
        "base_url": "http://alertmanager.monitoring.svc:9093",
        "bearer_token": "",
        "username": "",
        "password": ""
      }
    }
  ]
}

Credentials

Alertmanager itself often has no built-in API key UI. Credentials usually come from the reverse proxy or basic auth in front of the instance. Prefer a read-only token or user — OpenSRE only reads alerts and silences during investigations.

Investigation tools

Tool	What it does
alertmanager_alerts	Queries /api/v2/alerts for firing, silenced, and inhibited alerts
alertmanager_silences	Queries /api/v2/silences for active silences

alertmanager_alerts

Typical uses:

• Discover other alerts firing at the same time as the triggering alert
• Check whether the triggering alert is already silenced or inhibited
• Understand blast radius from active alert labels
• Correlate Prometheus alerts (OOM, latency, error-rate) into one timeline

The alertname label from the incoming alert is used as a filter by default so results stay scoped to the incident.

alertmanager_silences

Typical uses:

• See whether a noisy alert was silenced on purpose
• Surface maintenance windows that overlap the incident
• Avoid false root-cause conclusions from suppressed alerts

Verify

opensre integrations verify alertmanager

Expected output:

Service: alertmanager
Status:  passed
Detail:  Connected to Alertmanager at http://alertmanager.monitoring.svc:9093; cluster status: ready.

Local Docker verification

Start a disposable Alertmanager without external credentials:

docker run --rm --detach --name opensre-alertmanager \
  --publish 127.0.0.1:9093:9093 \
  prom/alertmanager:v0.28.1 || exit 1
for attempt in $(seq 1 30); do
  curl --fail --silent http://127.0.0.1:9093/-/ready && break
  if [ "$attempt" -eq 30 ]; then
    docker logs opensre-alertmanager
    exit 1
  fi
  sleep 1
done

Seed one firing alert and one active silence so both investigation tools return real data:

curl --fail --silent --request POST \
  --header "Content-Type: application/json" \
  --data '[{"labels":{"alertname":"OpenSREHighLatency","service":"checkout","severity":"critical"},"annotations":{"summary":"Checkout p99 latency exceeds 2 seconds"}}]' \
  http://127.0.0.1:9093/api/v2/alerts

starts_at=$(uv run python -c 'from datetime import UTC, datetime; print(datetime.now(UTC).isoformat())')
ends_at=$(uv run python -c 'from datetime import UTC, datetime, timedelta; print((datetime.now(UTC) + timedelta(hours=1)).isoformat())')
curl --fail --silent --request POST \
  --header "Content-Type: application/json" \
  --data "{\"matchers\":[{\"name\":\"alertname\",\"value\":\"OpenSREMaintenance\",\"isRegex\":false}],\"startsAt\":\"${starts_at}\",\"endsAt\":\"${ends_at}\",\"createdBy\":\"opensre-local-verification\",\"comment\":\"Disposable local verification silence\"}" \
  http://127.0.0.1:9093/api/v2/silences

From a source checkout, run uv run opensre integrations setup alertmanager. Enter http://127.0.0.1:9093 as the URL and select None for authentication, then verify the connection:

uv run opensre integrations verify alertmanager

The verifier should report cluster status ready. Exercise both tools through an agent turn:

uv run opensre ask \
  --allowed-tool alertmanager_alerts \
  --allowed-tool alertmanager_silences \
  "Report the firing Alertmanager alert and the active silence."

The result should include the OpenSREHighLatency alert and the OpenSREMaintenance silence. Stop the disposable instance when finished:

docker stop opensre-alertmanager

This unauthenticated instance binds only to loopback and is for local verification only.

Troubleshooting

Symptom	Fix
Status: missing	Set ALERTMANAGER_URL or run opensre integrations setup alertmanager
Connection refused	Confirm the URL is reachable from this host; check firewall rules
401 Unauthorized	Supply a bearer token or basic auth credentials (not both)
Both auth methods set	Clear either ALERTMANAGER_BEARER_TOKEN or the basic-auth pair
SSL error	Trust the CA cert, or use an http:// URL for internal instances
Empty alert list	Normal when nothing is firing — check the Alertmanager UI

Security

• Prefer a read-only reverse-proxy token — OpenSRE never writes to Alertmanager during investigations.
• Store credentials in .env or your secret manager — not in source control.
• For internal Kubernetes deployments, prefer network isolation over exposing credentials.