# How to Query Finelog for CPU and Memory Profile Captures

> Learn to query Finelog for CPU and memory profile captures using SQL on iris.task and iris.profile tables. Retrieve memory_peak_mb and cpu_time_sec metrics easily.

- Repository: [The Marin Project/marin](https://github.com/marin-community/marin)
- Tags: how-to-guide
- Published: 2026-08-29

---

**Use the `finelog query` CLI with standard SQL against the `iris.task` or `iris.profile` tables to retrieve `memory_peak_mb` and `cpu_time_sec` metrics.**

Finelog is the telemetry and profiling storage system in the marin-community/marin repository. It stores per-task CPU and memory data in predefined SQL tables that you query through a command-line interface. Understanding how to extract these metrics is essential for debugging performance bottlenecks in distributed task execution.

## Finelog Data Model for Profiling

Finelog organizes profiling data into two primary tables. Choosing the correct table determines whether you get aggregate summaries or time-series snapshots.

### The `iris.task` Table

The `iris.task` table stores one row per task execution with cumulative metrics. This is the default source for high-level performance analysis.

| Column | Type | Purpose |
|--------|------|---------|
| `task_id` | string | Unique task identifier |
| `attempt_id` | string | Execution attempt for retries |
| `memory_peak_mb` | float | Maximum memory observed (MiB) |
| `cpu_time_sec` | float | Cumulative CPU seconds consumed |

### The `iris.profile` Table

The `iris.profile` table contains periodic snapshots during task execution. Query this for fine-grained temporal analysis.

| Column | Type | Purpose |
|--------|------|---------|
| `profile_ts` | timestamp | When the sample was taken |
| `cpu_time_sec` | float | Cumulative CPU time at sample |
| `memory_peak_mb` | float | Memory usage at sample |

The schema definitions for both tables are located in [`lib/finelog/src/finelog/schema.py`](https://github.com/marin-community/marin/blob/main/lib/finelog/src/finelog/schema.py) as implemented in marin-community/marin.

## Basic Query Syntax

The Finelog CLI automatically handles authentication through cached Iris IAP credentials. The general invocation pattern is:

```bash
uv run finelog query <cluster> '<SQL>'

```

- `<cluster>` — Finelog deployment name (e.g., `marin`, `cw-us-east-08a`)
- `<SQL>` — Single-quoted SQL statement

For multiline queries, pass SQL via stdin:

```bash
uv run finelog query marin <<'EOF'
SELECT task_id, memory_peak_mb
FROM "iris.task"
LIMIT 5
EOF

```

The CLI implementation resides in [`lib/finelog/src/finelog/deploy/cli.py`](https://github.com/marin-community/marin/blob/main/lib/finelog/src/finelog/deploy/cli.py) according to the marin-community/marin source code.

## Querying Memory Peak Captures

To find the highest memory usage per task execution, aggregate `memory_peak_mb` from `iris.task`:

```bash
uv run finelog query marin '
SELECT task_id,
       attempt_id,
       max(memory_peak_mb) AS peak_mib
FROM   "iris.task"
GROUP BY task_id, attempt_id
ORDER BY peak_mib DESC
'

```

This query groups by both `task_id` and `attempt_id` to handle task retries correctly. The `max()` aggregation ensures you capture the worst-case memory consumption for each attempt.

### Filtering by Task Prefix

Add a `WHERE` clause to narrow results to specific task families:

```bash
uv run finelog query cw-us-east-08a <<'SQL'
SELECT task_id,
       attempt_id,
       max(memory_peak_mb) AS peak_mib
FROM   "iris.task"
WHERE  task_id LIKE '/power/example/%'
GROUP BY task_id, attempt_id
ORDER BY peak_mib DESC
LIMIT 10
SQL

```

Always include time bounds or `LIMIT` clauses to prevent full-table scans that trigger timeouts.

## Querying CPU Time Captures

Finelog does not store instantaneous CPU percentage. Instead, it tracks cumulative `cpu_time_sec`—the total processor seconds consumed by a task.

Retrieve total CPU time per execution:

```bash
uv run finelog query marin '
SELECT task_id,
       attempt_id,
       max(cpu_time_sec) AS cpu_seconds
FROM   "iris.task"
GROUP BY task_id, attempt_id
ORDER BY cpu_seconds DESC
'

```

The `max()` aggregation is technically redundant for `iris.task` (each row represents one attempt), but it defends against duplicate rows during data backfills.

### Time-Series CPU Analysis

For per-sample CPU progression, query `iris.profile`:

```bash
uv run finelog query marin '
SELECT task_id,
       attempt_id,
       profile_ts,
       cpu_time_sec,
       memory_peak_mb
FROM   "iris.profile"
WHERE  task_id = '/power/example/run-42'
  AND  profile_ts >= now() - INTERVAL '5 minutes'
ORDER BY profile_ts DESC
LIMIT 20
'

```

This reveals how CPU time accumulates over the task lifetime. Subtract successive `cpu_time_sec` values to derive per-interval CPU usage.

## Performance Best Practices

Efficient Finelog queries require attention to data volume and partitioning.

### Always Bound Time Ranges

Unbounded queries scan entire namespaces and frequently timeout. See the "Diagnosing query latency" section in [`lib/finelog/OPS.md`](https://github.com/marin-community/marin/blob/main/lib/finelog/OPS.md) (lines 73-82) for troubleshooting guidance.

**Bad:**

```bash
uv run finelog query marin 'SELECT * FROM "iris.profile"'

```

**Good:**

```bash
uv run finelog query marin '
SELECT *
FROM   "iris.profile"
WHERE  profile_ts >= now() - INTERVAL '1 hour'
'

```

### Discover Available Namespaces

List all queryable tables before writing queries:

```bash
uv run finelog namespaces marin

```

This command is documented in [`lib/finelog/OPS.md`](https://github.com/marin-community/marin/blob/main/lib/finelog/OPS.md) (lines 54-60) and helps verify table names against your target cluster's schema.

### Choose Output Formats

- Default: JSONL (machine-parseable)
- Human-readable: `--format table`

```bash
uv run finelog query marin 'SELECT * FROM "iris.task" LIMIT 3' --format table

```

## How Profiling Data Is Produced

Understanding data provenance helps interpret query results. Iris jobs emit profiling rows through [`lib/iris/scripts/job_profile_summary.py`](https://github.com/marin-community/marin/blob/main/lib/iris/scripts/job_profile_summary.py), which writes to Finelog during task execution. The Iris IAP authentication flow described in [`lib/finelog/OPS.md`](https://github.com/marin-community/marin/blob/main/lib/finelog/OPS.md) (lines 4-31) and [`lib/iris/OPS.md`](https://github.com/marin-community/marin/blob/main/lib/iris/OPS.md) enables secure CLI access to these records.

## Summary

- **Query `iris.task`** for per-execution aggregates of `memory_peak_mb` and `cpu_time_sec`
- **Query `iris.profile`** for time-series snapshots with `profile_ts` granularity
- **Use `uv run finelog query <cluster> '<SQL>'`** with stdin redirection for multiline statements
- **Always filter by time** to avoid timeouts on large namespaces
- **Reference [`lib/finelog/OPS.md`](https://github.com/marin-community/marin/blob/main/lib/finelog/OPS.md)** for authentication, query patterns, and performance diagnostics

## Frequently Asked Questions

### What is the difference between `iris.task` and `iris.profile`?

`iris.task` contains one row per task attempt with final cumulative metrics. `iris.profile` contains multiple rows per task—periodic samples captured during execution. Use `iris.task` for summary analysis and `iris.profile` for temporal debugging.

### Why doesn't Finelog store CPU percentage?

CPU percentage is a derived metric requiring sampling intervals and host attribution. Finelog stores `cpu_time_sec` as a cumulative counter, which is monotonic, mergeable across distributed workers, and independent of clock skew. Calculate percentage by differencing consecutive samples and dividing by wall-clock time.

### How do I handle authentication errors with `finelog query`?

The CLI retrieves Iris IAP credentials from your local cache. Run through the Iris authentication flow documented in [`lib/finelog/OPS.md`](https://github.com/marin-community/marin/blob/main/lib/finelog/OPS.md) (lines 4-31) and [`lib/iris/OPS.md`](https://github.com/marin-community/marin/blob/main/lib/iris/OPS.md) before issuing queries. Cached credentials typically expire after 24 hours.

### Can I join `iris.task` and `iris.profile` in a single query?

Yes—both tables share `task_id` and `attempt_id` keys. However, cross-table joins can be expensive. Prefer filtering each table independently, then correlating results in your analysis code to reduce query latency and resource consumption.