Remote runtime investigation

Investigate a deployed service using deployment status, logs, health checks, and the RCA pipeline.

Investigate problems on a deployed service by combining live signals (deployment status, logs, health) with OpenSRE’s standard RCA pipeline.

What changed

Earlier docs described:

opensre investigate --service <name>
opensre remote
opensre remote ops status
opensre investigate --service <name> --slack-thread CHANNEL/TS

Those commands were removed. There is no opensre remote CLI (do not confuse this with opensre remote-sync, which syncs sessions and memory to object storage).

The same goals are covered by the paths below.

Overview

To investigate a deployed service today:

1. Connect a cloud or deploy integration (Railway is the primary path for deploy inspection).
2. Gather live signals (deployment status, logs, health).
3. Run the standard RCA pipeline with opensre investigate and an alert payload that includes those signals — or ask the interactive shell to investigate in plain language so it can call the tools for you.

The RCA report shape is the same as opensre investigate -i <alert-file>.

Prerequisites

1. Deploy OpenSRE (or your app) so there is a live service to inspect. See Deployment.

2. Connect Railway (or another cloud integration you use):

   npm install -g @railway/cli
   railway login
   opensre integrations setup railway
   opensre integrations verify railway
   

   Optional env vars (also documented on Railway):

   RAILWAY_TOKEN=
   RAILWAY_PROJECT=project-id-or-name
   RAILWAY_SERVICE=service-id-or-name
   RAILWAY_ENVIRONMENT=production
   RAILWAY_PATH=railway
   

3. Configure an LLM (LLM_PROVIDER and the matching API key). See LLM providers.

Gather live signals

Deployment status (Railway)

Use the Railway integration tools (from the interactive shell, or as evidence during an investigation):

Tool	What it does	Confirmation
inspect_railway_deployment	Latest successful deployment and source commit metadata	No
redeploy_railway_service	Request a redeploy	Yes (confirm=true)

When defaults are configured, no parameters are required. Otherwise pass project, service, and environment. Full details: Railway.

Recent logs

Pull logs from the observability or platform tools you have connected (Datadog, Grafana, Railway CLI, and so on). Include the relevant lines in the alert payload or let the investigation agent call those tools.

Earlier remote-ops docs fetched about the last 100 log lines via a Railway ops provider. That dedicated ops layer is gone; use the integration tools and your log sources instead.

Health probe

Probe the service the same way you would for a hosted OpenSRE gateway:

curl https://<host>/ok
# or
curl https://<host>/health

Include the response (status code and body) in your alert context when it helps the investigation.

Run the RCA pipeline

From an alert file

opensre investigate -i alert.json
opensre investigate alert.json
opensre investigate -i alert.json -o ./rca.json
opensre investigate -i alert.json --evaluate

Other input modes:

opensre investigate --input-json '{"alert_name":"…","message":"…"}'
opensre investigate --interactive
opensre investigate --print-template generic

Build the alert JSON yourself (or start from --print-template) and include deployment status, log excerpts, and health results under fields such as message, raw_alert, or custom context keys your team uses.

From the interactive shell

opensre

Then describe the incident in plain language (for example: “inspect the latest Railway deploy for api-backend and investigate the error spike”). The agent can call inspect_railway_deployment and other tools, then run the investigation pipeline.

Slack thread context

The old --slack-thread CHANNEL/TS flag on investigate is gone. Slack thread context still works through:

1. Investigation / chat tool replay_slack_thread_locally with a thread_ref in CHANNEL/TS format.
2. Gateway Slack — when OpenSRE runs in Slack, thread history can be seeded automatically for that conversation.

Requirements (unchanged):

• Set SLACK_BOT_TOKEN (xoxb-…).
• The bot needs channels:history and groups:history for the channel you read.
• Get CHANNEL/TS from Slack’s “Copy link to message” — the last two path segments of the link.

Thread messages, users, timestamps, and reactions are fetched via Slack’s conversations.replies API. If the fetch fails (bad token, wrong channel, network error), the tool returns an error; the investigation can continue without that thread.

Example token setup:

export SLACK_BOT_TOKEN=xoxb-...

Then ask the shell to replay the thread, or include thread text in your alert payload after fetching it.

Investigate input rules

These input modes are mutually exclusive — use one path per run:

Mode	Flag / argument
Alert file	-i / --input, or a positional path
Inline JSON	--input-json
Paste in terminal	--interactive
Print a starter template	--print-template

Optional with any investigation run: -o / --output (write JSON) and --evaluate (LLM judge against a scoring_points rubric).

Extending beyond Railway

There is no longer a RemoteOpsProvider class under infra/deployment/remote/ops.py. That package was removed.

To support another deploy platform:

• Prefer a normal integration under integrations/<vendor>/ with tools (inspect deploy, fetch logs, and so on). See adding-tools-and-integrations.md in the repository.
• Or gather status/logs yourself and pass them into opensre investigate as alert context.

Other cloud pages in the docs (AWS, Kubernetes, Vercel, and so on) describe provider-specific tools you can combine the same way.

Known limitations

• No investigate --service / opensre remote / remote ops provider — use Railway (or other) integration tools plus opensre investigate or the interactive shell.
• Railway is the main first-class deploy-inspect integration for this workflow today. Other providers are available as separate integrations, not through a shared remote-ops interface.
• Slack context is thread-scoped — replay_slack_thread_locally pulls one thread via CHANNEL/TS. It does not search Slack history or resolve linked runbooks. Gateway Slack may attach thread history for live chats.
• alert_source may be re-inferred by the LLM — in the extract-alert step, the model can set alert_source from log text (for example "datadog" if logs mention Datadog), which routes to provider-specific tools. That is intended.

Related pages

• Railway — setup, inspect, redeploy
• Deployment — host OpenSRE / health checks
• Investigation overview — RCA pipeline
• Remote sync — session/memory sync (unrelated)
• Messaging / Slack — Slack bot setup