How OpenSRE works, end to end

A shared execution signal for analysis and optimization

OpenSRE is built to make execution behavior visible in compute-intensive environments, without changing workloads or relying on what applications choose to report.

At a high level, OpenSRE works in three layers:

• OpenSRE/collect: an open-source eBPF agent that gathers execution signals from the host-layer
• OpenSRE/datalake: a shared execution view across pipelines and environments
• OpenSRE/tune and OpenSRE/sweep: use that signal to solve different problems

This page explains the architecture. Each product page goes deeper on its specific behavior.

What OpenSRE is made of

OpenSRE consists of three components with distinct responsibilities:

• OpenSRE/collect gathers execution signals directly from the operating system
• OpenSRE/tune uses those signals to analyze and optimize pipeline performance
• OpenSRE/sweep uses the same signals to uncover systemwide cloud waste

OpenSRE/collect is the foundation. OpenSRE/tune and OpenSRE/sweep are built on top of the execution signal it produces.

Architecture at a glance

OpenSRE’s data flow can be understood in four stages:

1. Attach

   OpenSRE/collect attaches non-intrusively to running processes and containers on a Linux host using eBPF, a Linux kernel technology for safe, low-overhead instrumentation. No code changes, container restarts, or application modifications are required.

2. Collect

   Execution events are captured at the kernel boundary, including CPU scheduling, memory activity, disk and network I/O, and process lifecycle events. Only relevant signals are selected through intelligent filtering rules.

3. Correlate

   Low-level events are mapped to higher-level execution context such as containers, tools, tasks, runs, and pipelines. This mapping uses kernel-native identifiers like PIDs, namespaces, and cgroups.

4. Stream

   Structured telemetry is batched and sent securely to OpenSRE’s backend, where it becomes available for analysis, visualization, and downstream products. Data is buffered locally and retried until successfully delivered.

This pipeline is continuous and designed to operate safely in production cloud compute environments.

The execution signal (single source of truth)

OpenSRE’s execution signal is a structured representation of what actually ran on the system.

It includes:

• CPU usage and scheduling behavior
• Memory allocation and pressure
• Disk and network I/O activity
• Process lifecycles and relationships
• Container and host context

It explicitly does not include:

• Application payloads or scientific input/output data
• Source code, function calls, or language-level execution traces
• Application- or domain-specific interpretation of what a command does

The execution signal is derived from kernel-level observation via eBPF, without application instrumentation or code changes. It serves as the shared input for both OpenSRE/tune and OpenSRE/sweep.

How correlation works

Raw kernel events are not useful on their own. OpenSRE/collect correlates them into meaningful execution context.

At a high level:

• Kernel events are associated with processes
• Processes are grouped by containers and cgroups
• Containers and processes are mapped to tools, tasks, runs, and pipelines

This correlation allows OpenSRE to answer questions such as:

• Which tool generated this I/O?
• Which task was idle during this period?
• Which pipeline run consumed these resources?

All correlation is derived from operating system identifiers and execution context, not from workflow-specific integrations.

Where OpenSRE/tune fits

OpenSRE/tune focuses on pipelines that already work, but are slow or inefficient.

Using the execution signal, OpenSRE/tune:

• Visualizes actual resource usage at the task and process level
• Identifies underutilization, contention, and bottlenecks
• Distinguishes compute-bound, memory-bound, and I/O-bound stages
• Produces evidence-based recommendations for right-sizing and optimization

OpenSRE/tune answers: “How do we make this pipeline faster and cheaper?”

OpenSRE/tune

Learn more about pipeline performance optimization

Where OpenSRE/sweep fits

OpenSRE/sweep focuses on systemwide cloud efficiency.

Using the same execution signal, OpenSRE/sweep:

• Scans cloud compute based on real execution activity
• Identifies idle time, unused capacity, and hidden inefficiencies
• Surfaces waste that does not appear in billing reports or dashboards
• Avoids predictive shutdown heuristics by relying on observed behavior

OpenSRE/sweep answers: “Where are we wasting cloud spend right now?”

OpenSRE/sweep

Learn more about cloud waste detection

Choose your path

Depending on your goal, you can go deeper in different directions:

OpenSRE/collect

Learn how execution signals are captured safely and efficiently at the kernel level.

OpenSRE/tune

Learn how OpenSRE turns execution data into pipeline performance insights and recommendations.

OpenSRE/sweep

Learn how OpenSRE uncovers cloud waste using real activity patterns.