# How the Agentsview Sync Engine Detects and Processes New Sessions

> Learn how the Agentsview sync engine detects new sessions using fsnotify and scans, then processes them via a concurrent pipeline for classification, deduplication, and SQLite bulk writes.

- Repository: [Kenn Software/agentsview](https://github.com/kenn-io/agentsview)
- Tags: how-to-guide
- Published: 2026-06-12

---

**The agentsview sync engine uses a hybrid detection strategy combining real-time filesystem watching via `fsnotify` and periodic full-tree scans to discover new sessions, then processes them through a concurrent pipeline that classifies, deduplicates, and bulk-writes data to SQLite.**

Agentsview is an open-source session aggregator for AI coding agents that automatically imports and indexes conversation history from tools like Claude, Codex, and Copilot. At its core, the sync engine ([`internal/sync/engine.go`](https://github.com/kenn-io/agentsview/blob/main/internal/sync/engine.go)) orchestrates the discovery and ingestion of new session files as they appear on disk. Understanding how this engine detects and processes sessions is essential for customizing agent directories or troubleshooting import issues.

## Detection Mechanisms: Real-Time Watching and Periodic Scans

The engine employs two complementary strategies to ensure no session goes undetected.

### Real-Time Filesystem Watching

A recursive `fsnotify` watcher ([`internal/sync/watcher.go`](https://github.com/kenn-io/agentsview/blob/main/internal/sync/watcher.go)) monitors all configured agent directories. When files are created, written, renamed, or removed, the watcher debounces these events (default 200ms) and invokes `Engine.SyncPaths` with the affected paths. The `Watcher.handleEvent` method (lines 81-99) records paths and triggers the callback only after the debounce interval stabilizes, preventing redundant processing during rapid file operations.

### Periodic Full-Tree Scans

On startup and at regular intervals, the engine executes `Engine.SyncAll` or `Engine.ResyncAll` to perform a complete discovery pass. This guarantees that sessions missed by the watcher—such as batch files added while the program was offline—are still imported. These full scans walk every configured directory in `Engine.AgentDirs()` and feed discovered files into the same processing pipeline as real-time events.

## The Session Processing Pipeline

Once paths are detected, the engine executes an eight-step pipeline:

1. **Path Classification**: `Engine.SyncPaths` receives changed paths and invokes `classifyPaths` (lines 70-84) to map each file to a `parser.DiscoveredFile` struct. The `classifyOnePath` function (lines 445-720) matches paths against supported agent layouts (Claude, Codex, Copilot, Gemini, OpenHands, etc.) to determine the canonical session file location.

2. **Deduplication**: `dedupeDiscoveredFiles` (lines 445-475) collapses duplicate discoveries where multiple side-car files reference the same session, ensuring only unique sessions enter the parsing queue.

3. **Concurrent Parsing**: `startWorkers` launches a bounded worker pool (`maxWorkers = 8`) that parses discovered files concurrently. Each worker extracts session metadata and conversation content according to the specific agent format.

4. **Bulk Database Writes**: Parsed results flow through `collectAndBatch` (lines 94-99) which accumulates records into batches of `batchSize = 100` before upserting into SQLite via `syncWriteDefault` or `syncWriteBulk`. This minimizes database lock contention during large imports.

5. **Skip-Cache Management**: Files that fail to parse are recorded in `skipCache` via `persistSkipCache` (lines 298-311). These paths are excluded from future processing until their modification time changes, preventing repeated parsing errors.

6. **Event Emission**: After successful completion, the engine calls `emit` (deferred closure in lines 80-90) to notify registered `Emitter` instances. The HTTP server ([`internal/server/server.go`](https://github.com/kenn-io/agentsview/blob/main/internal/server/server.go)) uses this to broadcast SSE events, triggering UI refreshes when new sessions become available.

## Implementation Example

To initialize the sync engine and filesystem watcher:

```go
engine := sync.NewEngine(db, sync.EngineConfig{
    AgentDirs: map[parser.AgentType][]string{
        parser.AgentClaude: {"/home/user/.agentsview/claude"},
        parser.AgentCodex:  {"/home/user/.agentsview/codex"},
    },
    Machine: "my-laptop",
})

watcher, _ := sync.NewWatcher(200*time.Millisecond,
    func(paths []string) { engine.SyncPaths(paths) },
    []string{".git", "node_modules"},
)

for _, dirs := range engine.AgentDirs() {
    for _, root := range dirs {
        watcher.WatchRecursive(root)
    }
}
watcher.Start()

// Trigger initial full scan
go func() {
    ctx := context.Background()
    engine.SyncAll(ctx, nil)
}()

```

## Summary

- The agentsview sync engine combines **fsnotify-based filesystem watching** with **periodic full scans** to detect new sessions reliably.
- Detected paths are **classified** against agent-specific layouts and **deduplicated** before processing.
- A **worker pool** (`maxWorkers = 8`) parses sessions concurrently, with results **bulk-written** to SQLite in batches of 100.
- Failed parses are tracked in a **skip cache** to avoid redundant processing until files change.
- Successful syncs emit events that trigger **SSE broadcasts** to update the web UI immediately.

## Frequently Asked Questions

### How does the sync engine handle files that change while being processed?

The watcher debounces events using a 200ms interval, and the engine checks file modification times against the skip cache before parsing. If a file changes during processing, the subsequent `fsnotify` event triggers a re-sync, and `persistSkipCache` invalidates the cached entry when the mtime differs.

### What happens if the watcher misses a file while agentsview is offline?

The periodic `Engine.SyncAll` scan runs on startup and at regular intervals to discover any sessions created while the watcher was inactive. This full-tree traversal ensures eventual consistency regardless of filesystem events missed during downtime.

### Can the sync engine handle custom agent directory structures?

Yes. The `classifyOnePath` function in [`engine.go`](https://github.com/kenn-io/agentsview/blob/main/engine.go) supports multiple agent layouts (Claude, Codex, Copilot, Gemini, OpenHands). You configure custom roots via `EngineConfig.AgentDirs`, and the classifier matches discovered files against known patterns for each agent type.

### Why does the engine use batch writes instead of individual transactions?

Individual SQLite writes would create lock contention and degrade performance when importing thousands of sessions. The `collectAndBatch` mechanism accumulates up to 100 records before executing `syncWriteBulk`, significantly improving throughput while maintaining transactional integrity.