Hermes log monitoring

Tail Hermes errors.log, classify incidents, alert on Telegram or Rocket.Chat; run offline classifier regression tests, then try a local watch demo.

Overview

OpenSRE can watch the log file written by a Hermes deployment (typically errors.log), turn new lines into structured incidents, optionally deduplicate and escalate them, and deliver alerts to Telegram or Rocket.Chat. This page is about that log tail + classifier + delivery path — separate from OpenSRE’s own runtime logs.

OpenSRE can watch the log file written by a Hermes deployment (typically errors.log), classify new lines into structured incidents, optionally deduplicate and escalate them, and deliver alerts to Telegram.

This page covers the log watch + classifier + Telegram path and the investigation tool get_hermes_logs. Configure the investigation tool with opensre integrations setup hermes and check the saved path with opensre integrations verify hermes.

Delivery uses the same credential model as the rest of OpenSRE, for either provider. Telegram: configure once with opensre integrations setup telegram — hermes watch resolves the token from the integration store, environment, or keyring; set TELEGRAM_BOT_TOKEN / TELEGRAM_DEFAULT_CHAT_ID directly, or pass --chat-id. Rocket.Chat: pass --provider rocketchat and configure via opensre integrations setup rocketchat or ROCKETCHAT_* env vars. See Telegram and Rocket.Chat.

Item	Default
Log file for hermes watch	~/.hermes/logs/errors.log (override with --log-path)
Log file for get_hermes_logs	$HERMES_LOG_PATH if set, else ~/.hermes/logs/errors.log

The watcher uses a rotation-safe tailer. If the file is missing at startup, it waits until it appears. Use --from-start only when you intentionally want to replay existing contents (default avoids flooding Telegram on restart).

HERMES_LOG_PATH is read by the get_hermes_logs tool (default path and path sandbox). Pass --log-path for opensre hermes watch when the log is not at the default location.

Prerequisites

• A Hermes deployment that writes an errors.log (or another log you point at with --log-path for hermes watch; HERMES_LOG_PATH is for the get_hermes_logs tool)
• Telegram configured for alert delivery (bot token and chat ID) — see Credentials and Telegram
• For --investigate: LLM and integrations configured as for any OpenSRE investigation (quickstart)

Setup

Configure the log file used by investigations:

uv run opensre integrations setup hermes
uv run opensre integrations verify hermes

Setup defaults to ~/.hermes/logs/errors.log and only saves a readable regular file. It also sets HERMES_LOG_PATH, which defines the tool’s default path and permitted custom-log directory.

To run the long-lived watcher, use:

uv run opensre hermes watch

Stop with Ctrl+C or SIGTERM (for example under systemd).

Common options

Flag	Purpose
--log-path PATH	Hermes log file (default ~/.hermes/logs/errors.log)
--chat-id ID	Overrides TELEGRAM_DEFAULT_CHAT_ID for this run
--cooldown-seconds N	Per-fingerprint cooldown before the same incident is sent again (default 300)
--from-start / --from-end	Replay from the beginning, then tail (--from-start); default is end-only
--investigate / --no-investigate	Run OpenSRE RCA for HIGH / CRITICAL and append the summary to Telegram
--correlate / --no-correlate	Route through the correlator (dedup, escalation, routing). Default: on
--dedup-window-seconds	Correlator dedup window when --correlate is on
--escalation-threshold / --escalation-window-seconds	Repeat-hit escalation knobs when --correlate is on

Environment variables

Variable	Role
TELEGRAM_BOT_TOKEN	Bot token, unless already configured via opensre integrations setup telegram (see Telegram)
TELEGRAM_DEFAULT_CHAT_ID	Default destination when --chat-id is omitted
HERMES_LOG_PATH	Default path and path sandbox for get_hermes_logs (absolute path if Hermes writes elsewhere)
OPENSRE_HERMES_INVESTIGATE	If 1 / true / yes / on, enables investigation when the CLI does not pass --investigate / --no-investigate

Watch with full OpenSRE RCA (--investigate)

For production on-call, run one long-lived watcher with the RCA bridge when you want LLM-backed summaries on serious incidents:

uv run opensre hermes watch --investigate

Configure LLM + integrations as for any OpenSRE investigation (quickstart). When the classifier emits HIGH or CRITICAL, each qualifying incident triggers run_investigation via the Telegram sink. Work runs on a bounded thread pool with a timeout so log tailing stays responsive. MEDIUM and lower stay on a lighter notification path by default.

For ad hoc Hermes log context during another investigation (for example Grafana-driven RCA), use get_hermes_logs below — it samples the log and does not replace a long-running opensre hermes watch.

Quick check with a bundled fixture

Point --log-path at a synthetic errors.log. Prefer a scenario with ERROR / CRITICAL lines so incidents fire under the default classifier — the live watcher does not read per-scenario scenario.yml thresholds:

uv run opensre hermes watch \
  --log-path tests/synthetic/hermes/002-gateway-systemd-crash-loop/errors.log \
  --from-start

Example Telegram output when replaying that fixture (critical exit, traceback, crash loop, systemd error):

--from-start replays the file, then keeps tailing. By default, watch only reads new lines — without --from-start, a static fixture produces no events until more lines are appended. Prefer a writable copy of the scenario log if you append lines without editing files under tests/.

Omit --investigate for a lightweight demo. With --investigate, the investigation pipeline (LLM) runs for HIGH / CRITICAL only. Correlator dedup can also reduce multiple deliveries (either provider) for the same fingerprint; check shutdown line hermes-watch: correlator metrics delivered=… — if delivered=0, the classifier did not emit any routable incidents for that run.

Credentials

Telegram uses the same credential model as the rest of OpenSRE. Configure once with opensre integrations setup telegram. hermes watch resolves the token from the integration store, environment, or keyring. You can also set TELEGRAM_BOT_TOKEN / TELEGRAM_DEFAULT_CHAT_ID, or pass --chat-id. See Telegram.

Flag	Purpose
--log-path PATH	Hermes log file (defaults to HERMES_LOG_PATH expansion or ~/.hermes/logs/errors.log)
--provider telegram|rocketchat	Delivery provider (default telegram)
--chat-id ID	Overrides the provider’s default destination (TELEGRAM_DEFAULT_CHAT_ID or ROCKETCHAT_DEFAULT_CHANNEL) for this run
--cooldown-seconds N	Per-fingerprint cooldown before the same incident is sent again (default 300)
--from-start	Replay the file from the beginning, then tail
--investigate / --no-investigate	Run an OpenSRE RCA for HIGH / CRITICAL incidents and append the summary to the delivered message
--correlate / --no-correlate	Route through the correlator (dedup, escalation, routing). Default: on
--dedup-window-seconds	Correlator dedup window when --correlate is on
--escalation-threshold / --escalation-window-seconds	Repeat-hit escalation knobs when --correlate is on

Investigation tools

During an investigation, the planner can call get_hermes_logs:

Variable	Role
TELEGRAM_BOT_TOKEN	Bot token, unless already configured via opensre integrations setup telegram (see Telegram)
TELEGRAM_DEFAULT_CHAT_ID	Default Telegram destination when --chat-id is omitted
ROCKETCHAT_*	Rocket.Chat token or webhook credentials, used when --provider rocketchat (see Rocket.Chat)
ROCKETCHAT_DEFAULT_CHANNEL	Default Rocket.Chat destination when --chat-id is omitted
HERMES_LOG_PATH	Default log path when --log-path is omitted
OPENSRE_HERMES_INVESTIGATE	If set to 1 / true / yes / on, enables investigation when the CLI does not pass --investigate / --no-investigate

Mode	Behavior
op="scan"	One-shot window of the last N lines; returns parsed records and incidents the classifier would emit on that window
op="tail"	Incremental, cursor-based reads for “what’s new since last poll?”

Paths are restricted to permitted directories (by default ~/.hermes, plus the parent of HERMES_LOG_PATH when set) so arbitrary file reads are blocked.

Classifier state is not persisted between tail calls. Each tool invocation uses a fresh classifier, so burst windows and traceback continuation state reset between calls. For accurate ongoing detection, use opensre hermes watch (long-lived classifier) rather than many separate tail calls.

Other Hermes investigation tools

This page focuses on log watch and get_hermes_logs. Under integrations/hermes, additional session-evidence tools are also available when a Hermes backend is wired for RCA fixtures, including: Configure LLM + integrations as for any OpenSRE investigation (quickstart). When the classifier emits HIGH or CRITICAL incidents, each qualifying incident triggers run_investigation via the sink, regardless of --provider. Work runs on a bounded thread pool with a timeout so log tailing stays responsive; MEDIUM and lower stay on a lighter notification path by default.

get_hermes_session_log, get_hermes_session_topology, get_hermes_provider_traffic, get_hermes_adapter_catalog, get_hermes_config, get_hermes_message_history, get_hermes_kv_cache_state, get_hermes_runtime_state, get_hermes_cron_state, get_hermes_orchestration_state, get_hermes_routing_decisions, get_hermes_memory_state, get_hermes_filesystem_state, get_hermes_audit_trail, get_hermes_approval_events, get_hermes_rbac_state, get_hermes_credential_state, get_hermes_workflow_run.

Troubleshooting

Why you might see no Telegram

Cause	Detail
warning_burst threshold	Needs five warnings from the same logger within 60 seconds (IncidentClassifier defaults). Several synthetic scenarios lower that in scenario.yml for pytest only; the CLI ignores that file. Example: 000-telegram-polling-conflict expects bursts of three — the suite passes, but hermes watch on that file emits no warning_burst. Use a hard-error log (like 002-*) or a real errors.log with enough warnings.
Correlator dedup	Multiple sends for the same fingerprint can be suppressed. Check the shutdown line hermes-watch: correlator metrics delivered=… — if delivered=0, the classifier did not emit any routable incidents.

Security

Paths for get_hermes_logs are restricted to permitted directories (by default ~/.hermes, plus the parent of HERMES_LOG_PATH when set) so arbitrary file reads are blocked.

Extras

Local demo: synthetic tests, then watch

1 — Offline classifier checks (pytest, no Telegram)

The Hermes synthetic suite feeds real-shaped errors.log slices through IncidentClassifier and asserts against answer.yml. Same loop as tests/synthetic/hermes/README.md:

uv run pytest tests/synthetic/hermes -q

Incident-identification RCA fixtures (session topology, runtime hangs, cron delivery, KV cache drift):

uv run pytest tests/synthetic/hermes_rca -q

All synthetic-marked tests under tests/synthetic/:

make test-synthetic

2 — Live watch on a fixture (Telegram required)

Use a scenario errors.log that emits incidents under default classifier settings (see Quick check with a bundled fixture). From the repo root:

export TELEGRAM_BOT_TOKEN=…
export TELEGRAM_DEFAULT_CHAT_ID=…
uv run opensre hermes watch \
  --log-path tests/synthetic/hermes/002-gateway-systemd-crash-loop/errors.log \
  --from-start

Synthetic regression suite

Contributors add scenarios under tests/synthetic/hermes/ (fixtures, scenario.yml, answer.yml). Layout, schema, and “adding a new scenario” steps are in tests/synthetic/hermes/README.md.

Further reading

• Implementation: integrations/hermes/ (tailer, parser, classifier, correlator, sinks, CLI wiring).
• Surface-attribution evaluation for contributors: docs/hermes_runbook.mdx (repo only; not on the public docs nav).
• If you also run OpenClaw alongside Hermes, that page covers the OpenClaw bridge (context lookup and RCA write-back).
• For false-positive risks when rules match both message and raw line text, see issue #1874.