AWS CloudTrail Skill

Analyze AWS CloudTrail events and trails with parallel execution and anti-hallucination guardrails.

Relationship to other AWS skills:

• aws-cloudtrail/ → CloudTrail-specific analysis (events, trails, insights)
• aws/ → "How to execute" (parallel patterns, throttling, output format)

CRITICAL: Parallel Execution Requirement

ALL independent operations MUST run in parallel using background jobs (&) and wait.

#!/bin/bash
export AWS_PAGER=""

for trail in $trails; do
  get_trail_status "$trail" &
done
wait

Helper Functions

#!/bin/bash
export AWS_PAGER=""

# List trails
list_trails() {
  aws cloudtrail describe-trails \
    --output text \
    --query 'trailList[].[Name,IsMultiRegionTrail,IsOrganizationTrail,S3BucketName,HomeRegion,HasCustomEventSelectors]'
}

# Get trail status
get_trail_status() {
  local trail_name=$1
  aws cloudtrail get-trail-status --name "$trail_name" \
    --output text \
    --query '[IsLogging,LatestDeliveryTime,LatestDeliveryError,LatestNotificationTime,LatestNotificationError]'
}

# Lookup events (last N hours)
lookup_events() {
  local attribute_key=$1 attribute_value=$2 hours=${3:-24}
  local end_time start_time
  end_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
  start_time=$(date -u -d "$hours hours ago" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -v-${hours}H +"%Y-%m-%dT%H:%M:%SZ")
  aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey="$attribute_key",AttributeValue="$attribute_value" \
    --start-time "$start_time" --end-time "$end_time" \
    --max-results 20 \
    --output text \
    --query 'Events[].[EventTime,EventName,Username,Resources[0].ResourceName]'
}

# Get event selectors
get_event_selectors() {
  local trail_name=$1
  aws cloudtrail get-event-selectors --trail-name "$trail_name" \
    --output text \
    --query '[EventSelectors[].[ReadWriteType,IncludeManagementEvents,DataResources[].Type],AdvancedEventSelectors[].Name]'
}

# Get insight selectors
get_insight_selectors() {
  local trail_name=$1
  aws cloudtrail get-insight-selectors --trail-name "$trail_name" \
    --output text \
    --query 'InsightSelectors[].[InsightType]' 2>/dev/null
}

Common Operations

1. Trail Inventory and Health

#!/bin/bash
export AWS_PAGER=""
TRAILS=$(aws cloudtrail describe-trails --output text --query 'trailList[].Name')
for trail in $TRAILS; do
  {
    config=$(aws cloudtrail describe-trails --trail-name-list "$trail" \
      --output text --query 'trailList[].[Name,IsMultiRegionTrail,IsOrganizationTrail,S3BucketName,LogFileValidationEnabled]')
    status=$(aws cloudtrail get-trail-status --name "$trail" \
      --output text --query '[IsLogging,LatestDeliveryTime]')
    printf "%s\t%s\n" "$config" "$status"
  } &
done
wait

2. Recent API Activity by User

#!/bin/bash
export AWS_PAGER=""
USERNAME=$1
END=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
START=$(date -u -d "24 hours ago" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -v-24H +"%Y-%m-%dT%H:%M:%SZ")
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue="$USERNAME" \
  --start-time "$START" --end-time "$END" \
  --max-results 50 \
  --output text \
  --query 'Events[].[EventTime,EventName,EventSource,Resources[0].ResourceName]' | sort -k1

3. Security Event Investigation

#!/bin/bash
export AWS_PAGER=""
END=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
START=$(date -u -d "24 hours ago" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -v-24H +"%Y-%m-%dT%H:%M:%SZ")

# Check for console login events
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLogin \
  --start-time "$START" --end-time "$END" --max-results 20 \
  --output text \
  --query 'Events[].[EventTime,Username,Resources[0].ResourceName]' &

# Check for IAM changes
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventSource,AttributeValue=iam.amazonaws.com \
  --start-time "$START" --end-time "$END" --max-results 20 \
  --output text \
  --query 'Events[].[EventTime,EventName,Username]' &
wait

4. Resource Change Tracking

#!/bin/bash
export AWS_PAGER=""
RESOURCE_NAME=$1
END=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
START=$(date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -v-7d +"%Y-%m-%dT%H:%M:%SZ")
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue="$RESOURCE_NAME" \
  --start-time "$START" --end-time "$END" --max-results 50 \
  --output text \
  --query 'Events[].[EventTime,EventName,Username,EventSource]' | sort -k1

5. Event Selector Audit

#!/bin/bash
export AWS_PAGER=""
TRAILS=$(aws cloudtrail describe-trails --output text --query 'trailList[].Name')
for trail in $TRAILS; do
  {
    selectors=$(aws cloudtrail get-event-selectors --trail-name "$trail" \
      --output text \
      --query '[EventSelectors[].[ReadWriteType,IncludeManagementEvents],AdvancedEventSelectors[].Name]')
    printf "%s\t%s\n" "$trail" "$selectors"
  } &
done
wait

Anti-Hallucination Rules

1. lookup-events has 90-day limit - CloudTrail lookup-events API only covers the last 90 days of management events. For older events, query S3 directly or use Athena.
2. Management vs data events - Management events (API calls like CreateBucket) are logged by default. Data events (S3 object access, Lambda invocations) require explicit configuration.
3. CloudTrailEvent field is JSON string - The CloudTrailEvent field in lookup-events results is a JSON string, not parsed JSON. Use jq with fromjson to parse it.
4. Read-only vs write-only - Event selectors can filter by ReadWriteType: All, ReadOnly, WriteOnly. Check this before assuming all events are captured.
5. Organization trails - Organization trails log events for all accounts. But member account users may not have access to query the trail.

Output Format

Present results as a structured report:

Aws Cloudtrail Report
═════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Counter-Rationalizations

Shortcut	Counter	Why
"I'll skip discovery and check known resources"	Always run Phase 1 discovery first	Resource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"	Follow the full discovery → analysis flow	Quick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"	Audit configuration explicitly	Defaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"	Always check relevant metrics when available	API/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"	Try the command and report the actual error	Assumed permission failures prevent useful investigation; actual errors are informative

Common Pitfalls

• Event delivery delay: CloudTrail events can take up to 15 minutes to appear in lookup-events API.
• Log file validation: If LogFileValidationEnabled is false, log integrity cannot be verified. This is a security concern.
• Insight events: CloudTrail Insights detects unusual API activity. Must be explicitly enabled per trail. Costs extra.
• CloudWatch statistics syntax: Use spaces not commas: --statistics Average Maximum.
• S3 data events volume: Enabling S3 data events can generate massive volumes. Use advanced event selectors to filter by bucket/prefix.