How to Write Log Insights Queries for CloudWatch Log Analysis: A Complete Guide

Use CloudWatch Logs Insights' pipe-delimited query language—combining fields, filter, parse, stats, sort, and limit clauses—to extract, aggregate, and visualize log data directly from your AWS environment.

The aws/agent-toolkit-for-aws repository provides a definitive reference for building these queries, documenting every command and pattern you need to analyze logs efficiently. This guide walks through the core syntax, practical examples, and troubleshooting techniques found in the toolkit's observability skills.

Understanding the Core Query Syntax

CloudWatch Logs Insights uses a specialized SQL-like syntax that processes log data through a pipeline of commands. Each command is separated by a pipe character (|), allowing you to transform raw log entries into structured insights.

Selecting Fields with fields

The fields command defines which attributes appear in your output. You can reference system fields like @timestamp, @message, and @logStream, or extract custom fields from your log format.

fields @timestamp, @message, @logStream
| limit 20

As detailed in the log-insights.md reference, always specify your desired fields first to reduce data transfer and improve query performance.

Filtering Data with filter

The filter command applies boolean predicates to narrow your result set. You can use comparison operators (=, !=, <, >), pattern matching (like, =~), and logical operators (and, or).

fields @timestamp, @message
| filter @message like /ERROR/ and @timestamp > 1686816000
| sort @timestamp desc

Parsing Unstructured Data with parse

When logs contain embedded JSON or custom text formats, the parse command extracts structured fields using grok-style patterns or regular expressions.

fields @message
| parse @message "* * *" as ip, user, action
| filter action = "DELETE"
| stats count() by ip

This pattern creates new temporary fields (ip, user, action) that can be referenced in subsequent pipeline stages.

Aggregating with stats

The stats command performs calculations across your dataset, supporting functions like count(), sum(), avg(), min(), max(), and percentile(). Group results using the by clause.

fields durationMs, endpoint
| filter durationMs > 0
| stats avg(durationMs) as avgLatency, count() as requestCount by endpoint
| sort avgLatency desc

Practical Log Insights Query Examples

The following patterns demonstrate how to solve common operational questions using the exact syntax validated in the agent toolkit's reference files.

1. List the Latest Error Messages

Retrieve the most recent 20 error entries to start incident investigation.

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 20

2. Count Occurrences of Each Error Type

Extract the error level and message body, then aggregate to find frequency distribution.

fields @message
| parse @message "* *" as level, rest
| filter level = "ERROR"
| stats count() as errorCount by rest
| sort errorCount desc

3. Calculate Average Latency per API Endpoint

Analyze performance by computing average response times grouped by endpoint.

fields @timestamp, endpoint, durationMs
| filter durationMs > 0
| stats avg(durationMs) as avgLatency by endpoint
| sort avgLatency desc

4. Identify Top Traffic Sources by IP Address

Parse Apache-style access logs to find the five most active IP addresses.

fields @message
| parse @message "* * *" as ip, user, action
| stats count() as hits by ip
| sort hits desc
| limit 5

5. Join Request and Response Logs to Measure Latency

Use subqueries to correlate events across different log streams and calculate end-to-end latency.

# Subquery: get request timestamps

subquery request_logs
| fields @timestamp as reqTime, requestId
| filter @message like /REQUEST/
| stats min(@timestamp) as startTime by requestId

# Main query: correlate with responses

fields @timestamp, requestId, @message
| filter @message like /RESPONSE/
| stats max(@timestamp) as endTime by requestId
| join request_logs on requestId
| stats (endTime - startTime) / 1000 as latencySec by requestId
| sort latencySec desc
| limit 10

Advanced Query Patterns

Beyond basic filtering, CloudWatch Logs Insights supports complex analytical operations documented in the toolkit's advanced sections.

Joining Log Streams with join

The join command merges results from multiple log groups or subqueries based on common fields, enabling correlation analysis across distributed systems. As implemented in the agent toolkit examples, always ensure join keys are indexed or high-cardinality fields to prevent memory exhaustion.

Pattern Matching and Subqueries

Pattern commands (pattern) automatically cluster similar log messages, while subqueries allow you to nest queries for multi-stage analysis. These features are detailed in the SKILL.md file, which explains how to invoke these patterns within automated workflows.

Troubleshooting and Optimization

When queries return timeout errors or excessive results, consult the troubleshooting.md reference. Key optimization strategies include:

• Time-range scoping: Always specify the narrowest @timestamp range in your filter clause to reduce scanned data volume.
• Field pre-filtering: Use fields immediately after the implicit log source to drop unnecessary columns before aggregation.
• Limit early testing: Add limit 10 during query development to validate syntax before running full scans.

For visualization best practices, the dashboards.md file explains how to embed these queries into CloudWatch dashboards for continuous monitoring.

Summary

• CloudWatch Logs Insights uses a pipe-delimited syntax processed sequentially from fields through limit.
• Essential commands include fields for projection, filter for boolean filtering, parse for regex extraction, and stats for aggregation.
• Complex analysis is supported via join operations and subqueries to correlate events across log streams.
• Performance optimization requires strict time-range filtering and early field selection, as documented in the toolkit's troubleshooting guides.
• Source files log-insights.md, SKILL.md, troubleshooting.md, and dashboards.md in the aws/agent-toolkit-for-aws repository provide the authoritative reference for query syntax and integration patterns.

Frequently Asked Questions

What is the maximum execution time for a CloudWatch Logs Insights query?

CloudWatch Logs Insights queries time out after 15 minutes of execution. For large datasets, optimize by narrowing the time range in your filter clause or using sampled queries. The troubleshooting guide in the agent toolkit recommends testing with limit clauses first to validate performance before full scans.

How does the parse command differ from the filter command?

The parse command extracts new fields from existing log text using pattern matching, creating structure from unstructured data. The filter command removes rows from the result set based on boolean conditions. According to the log-insights.md reference, parse should precede filter when you need to filter based on extracted values.

Can I query multiple log groups in a single CloudWatch Logs Insights query?

Yes, you can select multiple log groups in the CloudWatch console or API, but each query executes against the union of those groups. For cross-log-group correlation, use the join command with subqueries, as demonstrated in the agent toolkit's latency analysis example, which matches requestId values across separate request and response logs.

Why does my query show "No results found" when I can see matching logs in the console?

This typically occurs when the time range selected in the query window does not overlap with the @timestamp values in your filter, or when field names are case-sensitive. The troubleshooting documentation advises verifying that your filter clause uses exact field names and that the query time range includes the log events.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}