Backfilling Signals in AgentsView: Process, Implementation, and Best Practices

AgentsView backfills session quality signals by scanning the SQLite database for stale sessions, recomputing signal values in-memory via RecomputeSignals, and persisting a completion marker to ensure idempotent retries.

AgentsView stores quality signals—such as tool-failure counts, edit-churn, and health scores—for every session in its SQLite database. When new signal fields are added through schema migrations or legacy sessions require processing, the system must backfill these values by recomputing them in bulk.

How the Backfill Process Works

The backfill is orchestrated by three core components defined in internal/db/signals.go and internal/sync/engine.go:

• BackfillSignals: Scans the database and manages the backfill loop.
• RecomputeSignals: Performs the pure in-memory signal calculation.
• MarkSignalsBackfillDone: Records completion to prevent redundant processing.

Detecting the Need for Backfill

The process begins by checking the stats table for the sentinel marker session_quality_signals_v1. In internal/db/signals.go, the code queries:

var done int
db.getWriter().QueryRow(
    `SELECT count(*) FROM stats WHERE key = ? AND value != 0`,
    signalsBackfillMarker,
).Scan(&done)

• If done == 0, no backfill has ever run.
• If done > 0, the system triggers a partial backfill by filtering on quality_signal_version < CurrentQualitySignalVersion.

Selecting Candidate Sessions

The function constructs a dynamic SQL query to fetch sessions requiring updates:

query := `SELECT id FROM sessions WHERE message_count > 0`
if done > 0 {
    query += ` AND quality_signal_version < ?`
    args = append(args, CurrentQualitySignalVersion)
}
rows, _ := db.getReader().QueryContext(ctx, query, args...)

Only sessions containing at least one message are selected. If a previous backfill marker exists, the process targets only sessions with stale signal versions.

Per-Session Signal Recomputation

For each candidate session ID, BackfillSignals invokes the provided computeFn. In production, this delegates to engine.RecomputeSignals from internal/sync/engine.go:

func(bCtx context.Context, id string) error {
    return engine.RecomputeSignals(bCtx, id)
}

RecomputeSignals performs the following steps:

1. Loads full session metadata via GetSessionFull.
2. Retrieves message history via GetAllMessages.
3. Executes computeSignalsAndSecrets to derive the SessionSignalUpdate struct and secret findings.
4. Persists results via UpdateSessionSignals and ReplaceSessionSecretFindings.

If any step fails, the error propagates back to BackfillSignals, which skips the completion marker to enable retry on the next startup.

Completion Markers and Idempotency

After processing all candidates successfully, BackfillSignals calls MarkSignalsBackfillDone, which upserts the marker:

func (db *DB) MarkSignalsBackfillDone() error {
    _, err := db.getWriter().Exec(
        `INSERT INTO stats (key, value) VALUES (?, 1)
         ON CONFLICT(key) DO UPDATE SET value = excluded.value`,
        signalsBackfillMarker,
    )
    return err
}

This design ensures partial runs are safe: if any session fails, the marker remains unset, and the next execution retry only processes remaining stale sessions.

Code Implementation and Key Functions

You can trigger a backfill manually using the database and engine instances:

func backfillAll(db *db.DB, eng *sync.Engine) error {
    return db.BackfillSignals(context.Background(),
        func(ctx context.Context, id string) error {
            return eng.RecomputeSignals(ctx, id)
        })
}

In the production server (cmd/agentsview/main.go), the backfill runs asynchronously after HTTP server startup:

go func() {
    if err := database.BackfillSignals(
        ctx,
        func(bCtx context.Context, id string) error {
            return engine.RecomputeSignals(bCtx, id)
        },
    ); err != nil && ctx.Err() == nil {
        log.Printf("signals backfill: %v", err)
    }
}()

Interaction with Full Resyncs

When a full resync rewrites every session through the inline-signal path, the code short-circuits unnecessary backfill work by pre-marking the completion marker in cmd/agentsview/main.go (lines 66-73). This prevents walking a large database after a clean resync has already computed current signal values.

Summary

• Backfilling signals in AgentsView recomputes quality metrics for legacy or migrated sessions using SQLite markers for state tracking.
• The BackfillSignals function in internal/db/signals.go orchestrates detection, selection, and batch processing via a user-supplied compute function.
• RecomputeSignals in internal/sync/engine.go handles the pure in-memory calculation of signal values and secret findings.
• The system uses the session_quality_signals_v1 marker in the stats table to ensure idempotent, retry-safe execution.
• Partial failures leave the marker unset, guaranteeing automatic retry on the next startup without duplicating work.
• Full resyncs can pre-mark completion to skip unnecessary backfill passes.

Frequently Asked Questions

What triggers a signal backfill in AgentsView?

A backfill triggers when the stats table lacks the session_quality_signals_v1 marker, or when existing sessions have a quality_signal_version lower than the current system version. This typically occurs after schema migrations add new signal fields or when legacy sessions have never been processed by the inline signal path.

Is the backfill process safe to interrupt and restart?

Yes. The backfill is idempotent because MarkSignalsBackfillDone only writes the completion marker after all sessions process successfully. If the process interrupts or fails on any session, the marker remains unset, and the next startup will retry only the remaining stale sessions without duplicating previous work.

How does AgentsView handle concurrent writes during backfill?

The backfill runs in a separate goroutine while the file-watcher and periodic sync continue writing new sessions. Because new writes use the normal incremental path and the backfill only targets existing stale sessions via version filtering, concurrent operations remain safe and consistent.

Can I manually trigger a backfill without restarting the server?

Yes. You can manually invoke BackfillSignals from any context with access to the database and engine instances. Pass a closure wrapping engine.RecomputeSignals as the compute function, and handle errors appropriately to retry or log failures without affecting the server's main operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}