# How Session Parsers Extract Data from Various Agent Formats in AgentsView

> Learn how session parsers extract data from various agent formats by streaming JSONL, using gjson for field extraction, and applying transformations for unified structs in AgentsView.

- Repository: [Kenn Software/agentsview](https://github.com/kenn-io/agentsview)
- Tags: how-to-guide
- Published: 2026-07-04

---

**Session parsers extract data from various agent formats** by streaming JSONL files line-by-line, extracting specific fields with `gjson`, and applying agent-specific transformations to populate unified `ParsedSession` and `ParsedMessage` structs.

The `agentsview` project normalizes conversation logs from dozens of AI coding agents into a queryable SQLite database. At the heart of this pipeline are specialized session parsers that handle the unique JSONL structures of each agent while conforming to a common extraction interface defined in [`internal/parser/types.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/types.go).

## The Parser Architecture: Registry and Provider Interface

The system uses a registry-based architecture to manage disparate agent formats. In [`internal/parser/types.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/types.go), the `Registry` slice defines an `AgentDef` for each supported agent, specifying metadata and extraction behavior.

```go
var Registry = []AgentDef{
    {Type: AgentClaude, DisplayName: "Claude Code", FileBased: true},
    {Type: AgentCopilot, DisplayName: "Copilot", FileBased: true},
    // ... additional agents
}

```

The `FileBased` flag signals the sync engine to invoke a file-system parser. Each agent implements a **provider** that exposes the `ParseUploadedTranscript` method, which serves as the uniform entry point for extraction. For example, [`internal/parser/claude_provider.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/claude_provider.go) wraps the Claude-specific logic:

```go
func (p *claudeProvider) ParseUploadedTranscript(
    path, project, machine string,
) ([]ParseResult, error) {
    return claudeParseWithExclusions(path, project, machine)
}

```

Similarly, [`internal/parser/copilot_provider.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/copilot_provider.go) delegates to `parseCopilot`, ensuring every agent follows the same contract while allowing format-specific implementations.

## Line-by-Line Ingestion: The lineReader Utility

To handle arbitrarily large session files without exhausting memory, parsers use a reusable `lineReader` defined in [`internal/parser/linereader.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/linereader.go). This utility buffers input and safely accumulates lines that exceed the internal buffer size (up to 64 MiB), preventing crashes from malformed or extreme JSONL entries.

```go
lr := newLineReader(f, maxLineSize)
for {
    line, ok := lr.next()
    if !ok { break }
    // Process raw JSON line
}

```

This streaming approach ensures that multi-gigabyte session files from long-running agent conversations can be processed with constant memory overhead.

## JSON Field Extraction with gjson

Rather than unmarshaling entire JSON objects into Go structs, parsers extract only the required fields using the **gjson** library. This optimization avoids the overhead of parsing irrelevant agent-specific metadata.

In [`internal/parser/claude.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/claude.go), extraction looks like this:

```go
entryType := gjson.Get(line, "type").Str
if ts := extractTimestamp(line); !ts.IsZero() {
    // Process timestamp
}

```

Parsers pull out universal schema fields such as `id`, `type`, `timestamp`, `content`, `tool_use_id`, `model`, and `token_usage`, leaving the raw JSON structure otherwise intact.

## Agent-Specific Parsing Strategies

While the core extraction pattern is consistent, each agent requires specialized handling for its unique conversation model.

### Claude Code: DAG Construction and Tool Result Handling

The Claude parser in [`internal/parser/claude.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/claude.go) handles complex conversation trees. It builds a directed acyclic graph (DAG) from `uuid` and `parentUuid` fields to detect conversation forks and branches.

The implementation also manages **queued commands**—attachments that arrive while a tool call is running—and resolves file references using `persistedToolResultPathRe`. The function `claudeParseWithExclusions` orchestrates this logic, returning fully resolved `ParseResult` slices that include tool output inlined into the message stream.

### GitHub Copilot: Tool Usage and Ordinal Sorting

In [`internal/parser/copilot.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/copilot.go), the `parseCopilot` function handles Copilot's JSONL format, which emits separate records for user messages, assistant responses, and tool invocations. The parser aggregates **usage events** emitted by the Copilot extension and sorts messages by the `ordinal` field to reconstruct the correct chronological order regardless of file write sequence.

### OpenAI Codex: Session Index Resolution

The Codex parser demonstrates handling external metadata files. In [`internal/parser/codex.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/codex.go), the `ParseCodexSessionIndexTitles` function reads a companion `session_index.jsonl` file to map raw session IDs to human-readable titles, enriching the `ParsedSession` before persistence.

## From Raw Lines to Unified Schema: The ParseResult Pipeline

After processing all lines, parsers construct a `ParseResult` containing a `ParsedSession`, a slice of `ParsedMessage` objects, and optional `ParsedUsageEvent` entries. The sync engine then persists these to SQLite.

For token usage, parsers may emit explicit fields, but when absent, the generic helper `InferTokenPresence` (in [`internal/parser/types.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/types.go)) inspects the raw `token_usage` JSON to set boolean flags like `HasContextTokens` and `HasOutputTokens`.

## Implementation Example: Parsing Claude and Copilot Sessions

The following example demonstrates how to invoke the parsers directly for Claude and Copilot session files:

```go
package main

import (
    "fmt"
    "github.com/kenn-io/agentsview/internal/parser"
)

func main() {
    // Parse a Claude session file
    claude := &parser.ClaudeProvider{}
    results, err := claude.ParseUploadedTranscript(
        "/home/user/.claude/projects/myproject/2023-09-01-abc123.jsonl",
        "myproject",
        "my-machine",
    )
    if err != nil {
        panic(err)
    }
    
    for _, r := range results {
        fmt.Printf("Session %s – %d messages, %d tokens total\n",
            r.Session.ID, len(r.Messages), r.Session.TotalOutputTokens)
    }

    // Parse a Copilot session file
    copilot := &parser.CopilotProvider{}
    cpResults, _ := copilot.ParseUploadedTranscript(
        "/home/user/.copilot/session-state/2024-01-15-xyz.jsonl",
        "my-project",
        "my-machine",
    )
    fmt.Println("Copilot parsed", len(cpResults), "sessions")
}

```

## Summary

- **Registry-driven architecture**: [`internal/parser/types.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/types.go) defines agents via `AgentDef` structs, enabling a plugin-like system where `FileBased` flags trigger filesystem parsing.
- **Streaming ingestion**: The `lineReader` in [`internal/parser/linereader.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/linereader.go) processes JSONL files line-by-line with memory safety for large inputs.
- **Selective extraction**: Parsers use `gjson` to extract only necessary fields from each JSON line, avoiding expensive unmarshaling of full objects.
- **Agent-specific logic**: Claude handles DAGs and tool results ([`claude.go`](https://github.com/kenn-io/agentsview/blob/main/claude.go)), Copilot manages ordinals and usage events ([`copilot.go`](https://github.com/kenn-io/agentsview/blob/main/copilot.go)), and Codex resolves external index files ([`codex.go`](https://github.com/kenn-io/agentsview/blob/main/codex.go)).
- **Unified output**: All parsers return `ParseResult` structs containing `ParsedSession` and `ParsedMessage` data, normalized for SQLite storage.

## Frequently Asked Questions

### What is the role of the Provider interface in the parser architecture?

The **Provider interface** defines the contract that every agent parser must implement, specifically the `ParseUploadedTranscript` method. This abstraction allows the sync engine in [`internal/parser/discovery.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/discovery.go) to treat all agents uniformly—whether parsing Claude, Copilot, or Codex files—by calling the same method signature regardless of the underlying format.

### How does AgentsView handle extremely large session files without running out of memory?

The parsers use a custom **`lineReader`** utility located in [`internal/parser/linereader.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/linereader.go). This reader buffers input and accumulates oversized lines in a string builder rather than loading the entire file into memory, enabling safe processing of multi-gigabyte JSONL files with constant memory overhead.

### Why does the Claude parser use gjson instead of standard JSON unmarshaling?

The Claude parser uses **gjson** to perform selective field extraction because Claude session files contain complex nested structures including conversation DAGs and tool attachments. Extracting only the `uuid`, `parentUuid`, `type`, and `timestamp` fields with `gjson.Get` is significantly faster than unmarshaling the entire JSON object into a Go struct, especially when processing millions of lines.

### What distinguishes the Copilot parser's handling of conversation ordering?

The **Copilot parser** explicitly sorts messages by the `ordinal` field after extraction. Because Copilot writes conversation events asynchronously to the JSONL file, the physical line order may not match the logical conversation flow. The parser in [`internal/parser/copilot.go`](https://github.com/kenn-io/agentsview/blob/main/internal/parser/copilot.go) aggregates these events and sorts them by ordinal before constructing the final `ParsedMessage` slice.