How the Agentsview Sync Engine Detects and Processes New Sessions

The agentsview sync engine uses a hybrid detection strategy combining real-time filesystem watching via fsnotify and periodic full-tree scans to discover new sessions, then processes them through a concurrent pipeline that classifies, deduplicates, and bulk-writes data to SQLite.

Agentsview is an open-source session aggregator for AI coding agents that automatically imports and indexes conversation history from tools like Claude, Codex, and Copilot. At its core, the sync engine (internal/sync/engine.go) orchestrates the discovery and ingestion of new session files as they appear on disk. Understanding how this engine detects and processes sessions is essential for customizing agent directories or troubleshooting import issues.

Detection Mechanisms: Real-Time Watching and Periodic Scans

The engine employs two complementary strategies to ensure no session goes undetected.

Real-Time Filesystem Watching

A recursive fsnotify watcher (internal/sync/watcher.go) monitors all configured agent directories. When files are created, written, renamed, or removed, the watcher debounces these events (default 200ms) and invokes Engine.SyncPaths with the affected paths. The Watcher.handleEvent method (lines 81-99) records paths and triggers the callback only after the debounce interval stabilizes, preventing redundant processing during rapid file operations.

Periodic Full-Tree Scans

On startup and at regular intervals, the engine executes Engine.SyncAll or Engine.ResyncAll to perform a complete discovery pass. This guarantees that sessions missed by the watcher—such as batch files added while the program was offline—are still imported. These full scans walk every configured directory in Engine.AgentDirs() and feed discovered files into the same processing pipeline as real-time events.

The Session Processing Pipeline

Once paths are detected, the engine executes an eight-step pipeline:

1. Path Classification: Engine.SyncPaths receives changed paths and invokes classifyPaths (lines 70-84) to map each file to a parser.DiscoveredFile struct. The classifyOnePath function (lines 445-720) matches paths against supported agent layouts (Claude, Codex, Copilot, Gemini, OpenHands, etc.) to determine the canonical session file location.

2. Deduplication: dedupeDiscoveredFiles (lines 445-475) collapses duplicate discoveries where multiple side-car files reference the same session, ensuring only unique sessions enter the parsing queue.

3. Concurrent Parsing: startWorkers launches a bounded worker pool (maxWorkers = 8) that parses discovered files concurrently. Each worker extracts session metadata and conversation content according to the specific agent format.

4. Bulk Database Writes: Parsed results flow through collectAndBatch (lines 94-99) which accumulates records into batches of batchSize = 100 before upserting into SQLite via syncWriteDefault or syncWriteBulk. This minimizes database lock contention during large imports.

5. Skip-Cache Management: Files that fail to parse are recorded in skipCache via persistSkipCache (lines 298-311). These paths are excluded from future processing until their modification time changes, preventing repeated parsing errors.

6. Event Emission: After successful completion, the engine calls emit (deferred closure in lines 80-90) to notify registered Emitter instances. The HTTP server (internal/server/server.go) uses this to broadcast SSE events, triggering UI refreshes when new sessions become available.

Implementation Example

To initialize the sync engine and filesystem watcher:

engine := sync.NewEngine(db, sync.EngineConfig{
    AgentDirs: map[parser.AgentType][]string{
        parser.AgentClaude: {"/home/user/.agentsview/claude"},
        parser.AgentCodex:  {"/home/user/.agentsview/codex"},
    },
    Machine: "my-laptop",
})

watcher, _ := sync.NewWatcher(200*time.Millisecond,
    func(paths []string) { engine.SyncPaths(paths) },
    []string{".git", "node_modules"},
)

for _, dirs := range engine.AgentDirs() {
    for _, root := range dirs {
        watcher.WatchRecursive(root)
    }
}
watcher.Start()

// Trigger initial full scan
go func() {
    ctx := context.Background()
    engine.SyncAll(ctx, nil)
}()

Summary

• The agentsview sync engine combines fsnotify-based filesystem watching with periodic full scans to detect new sessions reliably.
• Detected paths are classified against agent-specific layouts and deduplicated before processing.
• A worker pool (maxWorkers = 8) parses sessions concurrently, with results bulk-written to SQLite in batches of 100.
• Failed parses are tracked in a skip cache to avoid redundant processing until files change.
• Successful syncs emit events that trigger SSE broadcasts to update the web UI immediately.

Frequently Asked Questions

How does the sync engine handle files that change while being processed?

The watcher debounces events using a 200ms interval, and the engine checks file modification times against the skip cache before parsing. If a file changes during processing, the subsequent fsnotify event triggers a re-sync, and persistSkipCache invalidates the cached entry when the mtime differs.

What happens if the watcher misses a file while agentsview is offline?

The periodic Engine.SyncAll scan runs on startup and at regular intervals to discover any sessions created while the watcher was inactive. This full-tree traversal ensures eventual consistency regardless of filesystem events missed during downtime.

Can the sync engine handle custom agent directory structures?

Yes. The classifyOnePath function in engine.go supports multiple agent layouts (Claude, Codex, Copilot, Gemini, OpenHands). You configure custom roots via EngineConfig.AgentDirs, and the classifier matches discovered files against known patterns for each agent type.

Why does the engine use batch writes instead of individual transactions?

Individual SQLite writes would create lock contention and degrade performance when importing thousands of sessions. The collectAndBatch mechanism accumulates up to 100 records before executing syncWriteBulk, significantly improving throughput while maintaining transactional integrity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}