AWS CloudTrail

Trace AWS configuration-change causality — who changed what, and when — during incidents

Overview

OpenSRE uses AWS CloudTrail to answer the first question of every cloud post-mortem: “who changed what, and when?” When an AWS alert fires, the planner can look up recent management events — IAM changes, security-group mutations, EKS/Lambda config updates, and resource deletions — scoped to a resource, a principal, or a time window.

CloudTrail lookups are read-only and routed through the shared aws_sdk_client allowlist, so the integration cannot mutate any resources.

Prerequisites

• AWS credentials configured per the AWS integration (role ARN recommended for verify) — CloudTrail reuses the same account credentials and region, so no extra setup is needed
• IAM permission for the single read-only CloudTrail action listed below

Setup

CloudTrail is account-wide, so the tool becomes available to the planner whenever the AWS integration is configured — there is nothing resource-specific to set up. The region comes from the AWS integration (or AWS_REGION, defaulting to us-east-1).

opensre integrations setup aws

Credentials

IAM permissions

The tool only needs one read-only CloudTrail action:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudtrail:LookupEvents"
      ],
      "Resource": "*"
    }
  ]
}

Attach this policy to the same IAM role or user already configured for the AWS integration. If you are already using the AWS managed ReadOnlyAccess policy, this action is already covered.

Execution identity: the AWS integration’s role_arn / credentials gate availability and supply the region, but the lookup itself runs through boto3’s standard credential chain (environment variables, shared config, or the host’s instance role) — the configured role is not assumed for the call. Ensure the identity the OpenSRE process runs as can perform cloudtrail:LookupEvents. This matches the other AWS tools (RDS/EKS).

Investigation tools

Tool	AWS API call	What it returns
lookup_cloudtrail_events	cloudtrail:LookupEvents	Recent management events — event name, time, source, acting username, affected resources, AWS region, source IP, and any error code.

Parameters

Parameter	Default	Description
resource_name	—	Filter to events touching a specific resource name/ARN (the most specific filter).
event_source	—	Filter by AWS service event source, e.g. iam.amazonaws.com, ec2.amazonaws.com.
username	—	Filter by the acting principal / IAM username.
region	us-east-1	AWS region to query.
duration_minutes	60	Look-back window. CloudTrail retains 90 days of history (the upper bound).
max_results	50	Maximum events to return (CloudTrail caps this at 50 per call).
next_token	—	Pagination token from a previous truncated response; pass it to fetch the next page.

CloudTrail’s LookupEvents API accepts only one filter attribute per call. When more than one filter is supplied, the tool sends the most specific one, in priority order: resource_name → username → event_source. With no filter, it returns recent account-wide events for the window.

CloudTrail returns at most 50 events per page. When more matching events exist, the response sets truncated: true and returns a next_token — pass it back via the next_token parameter to fetch the next page, so a busy account or wide window never silently drops events.

A single event can touch more resources than the transport layer returns inline (for example CreateTags across a fleet). When the affected-resources list for an event is trimmed, that event carries resources_truncated: true, signalling that the real blast radius is wider than the resources shown — so a large fan-out change is never silently understated.

Use cases

• Finding who modified an IAM policy, role, or security group just before an incident
• Tracing configuration changes to a specific resource (by resource name)
• Auditing every action taken by a principal (by username)
• Reviewing recent activity from a single AWS service (by event source)
• Establishing change causality at the start of a post-mortem

Verify

opensre integrations verify aws

There is no separate cloudtrail verify target. CloudTrail availability is gated on the AWS integration; the lookup itself uses the ambient credential chain at investigation time.

Troubleshooting

Symptom	Fix
AccessDenied on cloudtrail:LookupEvents	Add the IAM policy above to the identity the OpenSRE process runs as (ambient chain), not only a role assumed during verify.
No events returned	Widen duration_minutes, loosen the filter, or confirm you are querying the region where the activity occurred. Only management events are returned; data events are not.
ThrottlingException	CloudTrail LookupEvents is rate-limited to two requests per second per account per region. Retry with a narrower window.
Tool reports the wrong region	Set AWS_REGION, or check the region field on the configured AWS integration.

Security

• CloudTrail lookups are read-only and cannot mutate resources.
• Prefer least-privilege IAM with only cloudtrail:LookupEvents if you do not need broader AWS access.
• Store AWS credentials in .env or your secret manager — not in source control.