How to Debug the Managed-Workstream Ledger for Cross-Harness Continuity

The managed-workstream ledger is an append-only SQLite log that records every event from native harnesses, enabling different sessions to resume runs and maintain consistent transcripts across tool invocations.

The akitaonrails/ai-memory repository implements this ledger as a portable event store that bridges CLI tools, MCP servers, and custom plugins. When cross-harness continuity breaks—whether due to lost leases, duplicate events, or stalled checkpoints—you need systematic debugging techniques to inspect the raw SQLite tables, validate lease files, and verify event sequences. This guide walks through the exact source locations, diagnostic queries, and remediation steps required to restore ledger integrity.

Understanding the Managed-Workstream Ledger Architecture

The ledger operates as a durable event stream that persists in SQLite and synchronizes across process boundaries. Understanding its five-phase lifecycle is essential before debugging.

Core Components and Data Flow

According to the source code in crates/ai-memory-core/src/workstream.rs, the ledger relies on three primary structures:

• PrepareManagedRunRequest – Initiates a managed run by acquiring a lease and selecting (or creating) a workstream. Returns workstream_id and high-water marks (sync_after, sync_through).
• NewWorkstreamEvent – Wraps every native transcript event (messages, tool calls, checkpoints) into a JSON payload posted to /workstream/event.
• FinishManagedRunRequest – Signals completion, triggering event import, compaction records, and retention policy enforcement.

During execution, the native harness emits events through the hooks layer defined in crates/ai-memory-hooks/src/workstream.rs, which forwards them to the MCP server. The server writes these to the workstream_events table managed by crates/ai-memory-store/src/migrations.rs.

The SQLite Persistence Layer

Each ledger row corresponds to a NewWorkstreamEvent with the following critical columns:

• event_id – Must be deterministic to prevent duplicates
• workstream_id – Links events to a specific run stream
• kind – Event type (message, tool_call, tool_result, checkpoint, compaction)
• occurred_at – ISO 8601 timestamp for ordering
• metadata – JSON blob containing repo fingerprints and checkpoint data

The lease mechanism creates a temporary managed-workstream.md file on disk containing run_id, lease_owner, and expiration data. The server validates this lease before accepting events to prevent race conditions between harnesses.

Debugging Common Cross-Harness Continuity Issues

When workstreams fail to sync across sessions or show inconsistent state, investigate these four specific failure modes.

Missing Events and Lease Failures

Symptom: A run appears incomplete in the ledger despite the harness finishing successfully.

Likely Cause: The child process crashed before sending FinishManagedRunRequest, or the lease file was deleted prematurely, causing the server to reject final event imports.

Debug Steps:

1. Verify the lease file exists in the temporary run directory: managed-workstream.md
2. Check for a compaction event in the ledger (search event_kind = "compaction" in workstream_events)
3. Query the MCP admin endpoint /admin/workstream/leases for stale or orphaned lease records

If the lease exists but events are missing, the import phase likely failed. Check the MCP server logs for SQLite constraint violations during the FinishManagedRunRequest handler.

Duplicate Event Entries

Symptom: The same tool call or message appears multiple times in search results.

Likely Cause: The client retried a POST after a network timeout, or the event_id was not derived deterministically from the native record ID.

Resolution:

Query the SQLite database directly to confirm duplicates:

SELECT event_id, COUNT(*) AS dup
FROM workstream_events
WHERE workstream_id = 'your-workstream-id'
GROUP BY event_id
HAVING dup > 1;

Prune duplicates via the admin endpoint:

ai-memory admin workstream purge --id <workstream_id>

Alternatively, manually delete specific rows:

DELETE FROM workstream_events 
WHERE workstream_id = 'your-workstream-id' 
AND event_id = 'duplicate-id';

Orphaned Tool Calls

Symptom: tool_call events lack corresponding tool_result entries, leaving async operations hanging.

Debug Procedure:

1. Search the ledger for event_kind = "tool_call" without subsequent tool_result entries
2. Verify the harness correctly forwards results by inspecting crates/ai-memory-hooks/src/workstream.rs
3. Check that the native tool execution actually completed and emitted the result payload before the process exited

This often indicates the harness was terminated (SIGTERM) before the tool finished, or the hook implementation failed to capture STDERR/STDOUT properly.

Stalled Checkpoints

Symptom: The workstream refuses to advance past a specific point despite new events being emitted.

Root Cause: Checkpoint validation failed due to a repo_fingerprint mismatch between the checkpoint metadata and the PrepareManagedRunRequest.repo_fingerprint.

Fix:

Query the latest checkpoint:

SELECT metadata->>'repo_fingerprint' as fingerprint, 
       metadata->>'head' as git_head 
FROM workstream_events 
WHERE workstream_id = 'ws-01' 
AND kind = 'checkpoint' 
ORDER BY rowid DESC 
LIMIT 1;

Compare the output against your current repository state using git rev-parse HEAD. If mismatched, the ledger is protecting against cross-repository contamination—verify you are running the harness in the correct working directory.

Step-by-Step Debugging Procedures

Use this systematic checklist when investigating ledger inconsistencies:

1. Identify the target workstream – Run ai-memory admin workstream list and note the workstream_id

2. Inspect the lease file – Open managed-workstream.md in the temporary run directory and confirm run_id and lease_owner match expected values

3. Query raw events – Execute:

   ai-memory workstream-search --id <workstream_id> --raw

   Look for gaps in sync_after/sync_through sequences or missing event_id fields

4. Validate database integrity – Connect to SQLite and verify monotonic rowid increase:

   SELECT rowid, event_id, kind, occurred_at 
   FROM workstream_events 
   WHERE workstream_id = 'ws-01' 
   ORDER BY rowid;

5. Check compaction boundaries – Ensure a compaction event appears after event batches. Missing compaction indicates the ledger is stuck in an un-compacted state, potentially consuming excess disk space

6. Review admin logs – Query /admin/workstream/logs for lease acquisition errors or HTTP 409 conflict responses indicating duplicate event rejections

7. Enable debug logging – Re-run with verbose output to trace every HTTP request:

   RUST_LOG=debug ai-memory run -- <your-command>

8. Manual lease cleanup – If the lease is stale, release it via:

   curl -X POST http://127.0.0.1:49374/api/v1/admin/workstream/lease/release \
        -d '{"workstream_id":"ws-01","run_id":"run-abcde"}'

Diagnostic Code Examples

Manual Event Injection for Testing

Use this Rust snippet to reproduce events directly against the store API, bypassing the harness:

use ai_memory_core::{NewWorkstreamEvent, WorkstreamEventKind, AgentKind};
use ai_memory_store::StoreClient;

let event = NewWorkstreamEvent {
    event_id: "debug-123".into(),
    agent: AgentKind::Cli,
    native_session_id: "test-session".into(),
    source_record_id: None,
    kind: WorkstreamEventKind::Message,
    role: Some("assistant".into()),
    content: "Diagnostic message".into(),
    occurred_at: Some(chrono::Utc::now().to_rfc3339()),
    metadata: serde_json::json!({ "debug": true }),
};

let client = StoreClient::connect("http://127.0.0.1:49374").await?;
client.post_workstream_event(event).await?;

Live Ledger Tailing

Monitor events as they arrive in real-time:

RUST_LOG=debug ai-memory workstream-search \
    --id my-workstream \
    --tail

Lease File Inspection

Parse the lease metadata directly:

cat /tmp/ai-memory-run-$(uuid)/managed-workstream.md

# lease_owner: "ai-memory-cli:12345"

# run_id: "run-abcde"

# workstream_id: "ws-01"

# expires_at: "2026-09-01T12:00:00Z"

Administrative Purge

When corruption is irreparable, purge and restart:

ai-memory admin workstream purge --id <workstream_id>

Summary

• The managed-workstream ledger in crates/ai-memory-core/src/workstream.rs provides cross-harness continuity through an append-only SQLite event log
• Lease management via managed-workstream.md files prevents concurrent write conflicts; stale leases cause event import failures
• Duplicate events stem from non-deterministic event_id generation or HTTP retries—query workstream_events grouped by event_id to detect them
• Checkpoint stalls occur when repo_fingerprint in the checkpoint metadata mismatches the current repository state
• Debug tooling includes ai-memory workstream-search, raw SQLite queries against workstream_events, and MCP admin endpoints in crates/ai-memory-mcp/src/admin.rs

Frequently Asked Questions

How do I verify that my workstream ledger is consistent across two different harness sessions?

Query the workstream_events table for the specific workstream_id in both sessions using SELECT rowid, event_id, kind FROM workstream_events WHERE workstream_id = 'ws-01' ORDER BY rowid. The event_id sequence should be identical and monotonically increasing in both environments. If discrepancies exist, check for un-compacted events or lease conflicts in the MCP admin logs.

What causes the "lease already held" error when starting a managed run?

This error occurs when a previous run crashed without sending FinishManagedRunRequest, leaving the lease file managed-workstream.md orphaned on disk or a stale entry in the server's lease registry. Release the lease via the admin endpoint /admin/workstream/lease/release with the appropriate workstream_id and run_id, or wait for the lease expiration timestamp to pass.

Can I manually edit the SQLite database to fix corrupted ledger entries?

Yes, but exercise caution. Connect directly to the SQLite file used by crates/ai-memory-store and execute standard SQL UPDATE or DELETE statements against workstream_events. After manual edits, run ai-memory admin workstream purge to trigger a compaction, ensuring that the high-water marks in PrepareManagedRunResponse stay synchronized with the actual table state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}