# Writing Efficient CloudWatch Log Insights Queries with parse, filter, and stats Commands

> Learn to write efficient CloudWatch Log Insights queries by filter early, parse second, and aggregate last for maximum performance. Optimize your logs today.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: how-to-guide
- Published: 2026-06-29

---

**Filter early, parse second, and aggregate last to minimize data scanned and maximize query performance in CloudWatch Log Insights.**

CloudWatch Log Insights provides serverless analytics for log data, but query efficiency depends entirely on how you structure your pipeline. In the `aws/agent-toolkit-for-aws` repository, the observability skill documentation demonstrates how to combine **filter**, **parse**, and **stats** commands to analyze millions of log events without unnecessary overhead.

## Understanding the Core Operators

CloudWatch Log Insights processes queries through a pipeline that streams log events through three primary operators. According to the reference documentation in [`skills/core-skills/aws-observability/references/log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/log-insights.md), each operator serves a distinct purpose in the analytics workflow.

### The filter Command

The **filter** command keeps only events matching a Boolean expression, such as `status >= 500` or `@message like /ERROR/`. This operator acts as the first line of defense, discarding irrelevant data before it reaches downstream stages.

### The parse Command

The **parse** command extracts structured fields from the raw `@message` field using string patterns or regular expressions. This turns free-form log lines into queryable columns without requiring schema changes to your application.

### The stats Command

The **stats** command performs aggregations including `count()`, `sum()`, `avg()`, and `pct()` (percentiles), optionally grouping results with the `by` clause. This operator works on the in-memory stream to summarize large datasets in seconds.

## Why Operator Order Matters for Performance

The Log Insights query engine evaluates pipelines **row-by-row**, making operator sequence critical for efficiency. Process events in this order:

1. **filter** – Reduce the dataset size immediately by eliminating unmatched events.
2. **parse** – Extract fields only from the filtered subset.
3. **stats** – Aggregate the already-filtered and parsed data.

This sequence ensures that **parse** and **stats** never see irrelevant events, reducing memory usage and execution time. The `aws/agent-toolkit-for-aws` documentation emphasizes that this order-sensitive execution keeps queries cost-effective even when scanning millions of log lines.

## Practical Query Examples from the Agent Toolkit

The following examples from [`skills/core-skills/aws-observability/references/log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/log-insights.md) demonstrate the filter-parse-stats workflow using the AWS CLI.

### Example 1: Counting Errors per User

This query filters for ERROR messages, extracts user details, and counts occurrences per user:

```bash
aws logs start-query \
  --log-group-name /aws/lambda/YOUR_LOG_GROUP \
  --start-time $(date -d '-1h' +%s) \
  --end-time $(date +%s) \
  --query-string '
filter @message like /ERROR/ 
| parse @message "User * performed * on *" as user, action, resource 
| stats count() as errors by user
| sort errors desc
'

```

### Example 2: Calculating Latency Percentiles per Endpoint

This query uses regex patterns to extract numeric latency values and computes both average and p99 latency:

```bash
aws logs start-query \
  --log-group-name /aws/apigateway/YOUR_LOG_GROUP \
  --start-time $(date -d '-24h' +%s) \
  --end-time $(date +%s) \
  --query-string '
| parse @message /User (?<user>\w+) performed (?<action>\w+)/
| parse @message /latency_ms=(?<latency>\d+)/ 
| stats avg(latency) as avg_ms, pct(latency, 99) as p99_ms by endpoint
| sort p99_ms desc
'

```

### Example 3: Time-Series Request Volume by HTTP Method

This query buckets requests into 5-minute intervals to create a histogram of traffic patterns:

```bash
aws logs start-query \
  --log-group-name /aws/elasticloadbalancing/YOUR_LOG_GROUP \
  --start-time $(date -d '-2h' +%s) \
  --end-time $(date +%s) \
  --query-string '
| parse @message "method=* " as httpMethod
| stats count() as requests by bin(5m), httpMethod
| sort bin asc
'

```

## Key Reference Files in aws/agent-toolkit-for-aws

The Agent Toolkit provides comprehensive documentation for Log Insights optimization:

- **[`skills/core-skills/aws-observability/references/log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/log-insights.md)** – The canonical cheat-sheet containing dozens of ready-to-copy patterns for `parse`, `filter`, and `stats` usage.

- **[`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md)** – High-level overview of observability capabilities, showing how Log Insights fits into broader monitoring strategies.

- **[`skills/core-skills/aws-observability/references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/metrics.md)** – Details metric conventions such as `@duration` and `@billedDuration` that you can reference in `stats` aggregations.

- **[`skills/core-skills/aws-observability/references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/alarms.md)** – Documentation for pairing Log Insights-derived metrics with CloudWatch Alarms.

## Summary

- **Filter first** to eliminate irrelevant log events before processing, reducing downstream computation.
- **Parse second** to extract structured fields from the filtered subset using patterns or regular expressions.
- **Stats last** to aggregate data efficiently on the smallest possible dataset.
- Consult **[`skills/core-skills/aws-observability/references/log-insights.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/log-insights.md)** in the `aws/agent-toolkit-for-aws` repository for syntax reference and additional examples.
- Execute queries via the CloudWatch Console, AWS CLI (`aws logs start-query`), or programmatically through the SDK.

## Frequently Asked Questions

### What is the correct order of operations in CloudWatch Log Insights?

Place **filter** first to reduce the dataset, followed by **parse** to extract fields, and end with **stats** to aggregate results. The query engine processes events row-by-row, so each subsequent operator only sees data that passed through the previous stages.

### Why should I use parse instead of just filtering raw messages?

The **parse** command extracts structured fields from unstructured log lines, enabling precise aggregation and filtering beyond simple string matching. Without parsing, you cannot calculate averages, percentiles, or group by specific values embedded in log messages.

### How do I calculate percentiles like p99 in Log Insights?

Use the `pct(field, 99)` function within a **stats** command. For example: `stats pct(latency, 99) as p99_ms by endpoint`. This computes the 99th percentile latency for each endpoint in your logs.

### Can I use regular expressions with the parse command?

Yes. The **parse** command supports regex patterns using the syntax `parse @message /regex/`. You can use named capture groups like `(?<field_name>\d+)` to extract specific values into named columns for subsequent filtering or aggregation.