> ## Documentation Index
> Fetch the complete documentation index at: https://opensre.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Grafana Tempo

> Connect Grafana Tempo so OpenSRE can query distributed traces during investigations

OpenSRE uses Grafana Tempo to investigate trace-related alerts — searching spans by service, fetching full traces by ID, listing instrumented services, and filtering by error status or latency.

This integration talks to Tempo **directly** via its HTTP API. It does not require a Grafana instance or datasource proxy. If you run the full Grafana stack, the [Grafana](/grafana) integration already surfaces Tempo through the datasource proxy — use this integration when you run Tempo standalone.

## Prerequisites

* Grafana Tempo 1.4+
* Network access from the OpenSRE environment to your Tempo instance
* Auth credentials only if your deployment requires them (many run without auth behind a gateway)

## Setup

### Option 1: Interactive CLI

```bash theme={null}
opensre integrations setup tempo
```

You will be prompted for the Tempo URL and optional auth. Leave auth fields blank if your Tempo runs without authentication.

### Option 2: Environment variables

Add to your `.env`:

```bash theme={null}
TEMPO_URL=http://localhost:3200
TEMPO_API_KEY=<bearer-token>       # optional
TEMPO_USERNAME=<username>          # optional (basic auth)
TEMPO_PASSWORD=<token>             # optional (basic auth)
TEMPO_ORG_ID=<tenant-id>           # optional (X-Scope-OrgID for multi-tenant)
```

| Variable         | Default   | Description                                                    |
| ---------------- | --------- | -------------------------------------------------------------- |
| `TEMPO_URL`      | —         | **Required.** Tempo HTTP API base URL                          |
| `TEMPO_API_KEY`  | *(empty)* | Bearer token for auth                                          |
| `TEMPO_USERNAME` | *(empty)* | Username for basic auth                                        |
| `TEMPO_PASSWORD` | *(empty)* | Password for basic auth                                        |
| `TEMPO_ORG_ID`   | *(empty)* | Tenant ID sent as `X-Scope-OrgID` for multi-tenant deployments |

### Option 3: Persistent store

Integrations are automatically persisted to `~/.opensre/integrations.json`:

```json theme={null}
{
  "version": 1,
  "integrations": [
    {
      "id": "tempo-prod",
      "service": "tempo",
      "status": "active",
      "credentials": {
        "url": "http://localhost:3200",
        "api_key": ""
      }
    }
  ]
}
```

## Investigation tools

OpenSRE exposes a single `query_tempo` tool with an `action` parameter:

### search

Searches traces by service, span name, duration, and tags using TraceQL. Returns one summary row per trace.

```text theme={null}
query_tempo(action="search", service="checkout-service", min_duration_ms=500)
query_tempo(action="search", tags={"http.status_code": "500"})
```

| Parameter            | Description                       |
| -------------------- | --------------------------------- |
| `service`            | Filter by `resource.service.name` |
| `span_name`          | Filter by span name               |
| `min_duration_ms`    | Minimum trace duration            |
| `max_duration_ms`    | Maximum trace duration            |
| `tags`               | Key/value span attributes         |
| `time_range_minutes` | Lookback window (default 60)      |
| `limit`              | Max traces to return (default 20) |

### get\_trace

Fetches a full trace by ID and flattens its spans.

```text theme={null}
query_tempo(action="get_trace", trace_id="4f8c...e21")
```

### list\_services

Lists all services registered in Tempo.

```text theme={null}
query_tempo(action="list_services")
```

### list\_span\_names

Lists all span names registered in Tempo.

```text theme={null}
query_tempo(action="list_span_names")
```

## Verify

```bash theme={null}
opensre integrations verify tempo
```

Expected output:

```
Service: tempo
Status:  passed
Detail:  Connected to Grafana Tempo HTTP API (/api/search, /api/traces).
```

## Troubleshooting

| Symptom                | Fix                                                                                                 |
| ---------------------- | --------------------------------------------------------------------------------------------------- |
| **Connection refused** | Verify the URL and port. Default Tempo HTTP port is `3200`.                                         |
| **HTTP 401**           | Set `TEMPO_API_KEY` or `TEMPO_USERNAME`/`TEMPO_PASSWORD` if your deployment requires auth.          |
| **HTTP 404 on verify** | Check your Tempo version — the `/api/search/tags` endpoint requires Tempo 1.4+.                     |
| **No traces returned** | Confirm your services are sending traces to Tempo and the time range covers the period of interest. |
| **Multi-tenant 400**   | Set `TEMPO_ORG_ID` to the correct tenant ID.                                                        |

## Security best practices

* Use a **read-only** token or service account if your Tempo deployment supports auth.
* Store credentials in `.env`, never in code.
* Restrict network access to Tempo — OpenSRE only needs the HTTP API port (`3200` by default).