How ai-memory's Atomic Wiki Write Pipeline Interacts with the Git2 Watcher

The ai-memory atomic wiki write pipeline uses temporary file writes followed by atomic renames to ensure the git2 watcher only ever sees complete, valid pages, while explicit tempfile filtering prevents feedback loops.

ai-memory implements a single source of truth for knowledge: Markdown files stored in <data_dir>/wiki/. Every write to this filesystem passes through a centralized atomic pipeline that coordinates with a git2-based file watcher. Understanding this interaction is essential for developers extending the wiki or debugging indexing behavior in the akitaonrails/ai-memory repository.

Core Architecture of the Atomic Write Pipeline

The write pipeline lives in three coordinated modules: atomic.rs performs low-level filesystem operations, wiki.rs orchestrates the full transaction, and watcher.rs observes results without interfering.

Step-by-Step Pipeline Execution

The atomic wiki write pipeline executes seven distinct phases when persisting page content:

1. Temporary File Creation

The atomic::write_atomic function in crates/ai-memory-wiki/src/atomic.rs writes bytes to a tempfile prefixed with .ai-memory-tmp.. This occurs outside the final destination path, ensuring no observer sees partial data:

// From wiki.rs line 549
atomic::write_atomic(&abs, raw.as_bytes())?;

2. Atomic Rename to Final Path

After fsync flushes the buffer, the tempfile undergoes an atomic rename to the target location. POSIX filesystems guarantee this rename appears instantaneous—observers see either the old file or the new file, never a corrupted intermediate state.

3. Optional Admission Webhooks

Before the rename executes, write_page invokes any configured admission chain, allowing hooks to mutate frontmatter or abort the write:

// wiki.rs lines 92-99
self.admission_chain
    .as_ref()
    .map(|chain| chain.notify(&mut markdown))
    .transpose()?;

4. SQLite Store Synchronization

Held under the exclusive mutation_lock, the pipeline sends WriteCmd::UpsertPage to the single-writer SQLite actor (writer). This upserts the page row in the same logical transaction as the filesystem write, keeping disk and database synchronized:

// wiki.rs lines 510-566
self.writer.upsert_page(NewPage { /* ... */ }).await?;

How the Git2 Watcher Observes Changes

The git2 watcher continuously monitors <wiki_root> through filesystem events. Its interaction with the atomic write pipeline centers on two critical behaviors: tempfile filtering and re-indexing timing.

Ignoring Atomic Tempfiles

The watcher explicitly excludes its own temporary files from processing. This ignores_own_atomic_tempfiles logic prevents a destructive feedback loop where the watcher would re-index its own in-progress writes:

// watcher.rs (lines ~730-745)
fn is_pending_path(path: &PagePath) -> bool {
    path.as_str().starts_with(".ai-memory-tmp.")
}

// Event handler
if is_pending_path(&changed_path) {
    return; // Skip: this was our own atomic write
}
wiki.reindex_page(workspace_id, project_id, changed_path).await?;

Triggering Re-Index After Rename

When the watcher receives a rename event for a non-tempfile path, it invokes reindex_page at lines 1249-1265 of wiki.rs. Because the atomic rename has already completed, the re-index operation reads the fully persisted content:

// wiki.rs lines 1249-1265
pub async fn reindex_page(
    &self,
    workspace_id: Uuid,
    project_id: Uuid,
    path: PagePath,
) -> Result<Page, Error> {
    // Parses fresh Markdown and upserts to store
}

Lock Coordination and Concurrency Safety

Both the writer and watcher acquire the same mutation_lock (RwLock) before touching filesystem or store. This ensures that operations like project moves remain invisible to concurrent writers, maintaining consistency across the entire pipeline.

Optional Git Commit Integration

After batch writes complete, callers may invoke Wiki::commit_all (lines 222-226 of wiki.rs) to create a git commit through the embedded GitAdapter. The watcher operates independently—filesystem events drive its behavior, not git history:

// wiki.rs lines 222-226
if let Some(oid) = wiki.commit_all("Batch update")? {
    println!("Created git commit {}", oid);
}

Complete Usage Example

Putting all components together, a typical write-and-commit flow looks like this:

use ai_memory_wiki::atomic;
use ai_memory_wiki::models::{NewPage, Tier};
use serde_json::json;

// Prepare page data
let new_page = NewPage {
    workspace_id,
    project_id,
    path: page_path.clone(),
    title: "Example".into(),
    body: "Content".into(),
    tier: Tier::Semantic,
    frontmatter_json: json!({ "title": "Example" }),
    pinned: false,
    links: vec![],
    author_id: None,
    expires_at: None,
    entities: vec![],
};

// Step 1-2: Atomic filesystem write
let abs = wiki.abs_path(workspace_id, project_id, &page_path);
let raw = emit(&Markdown { frontmatter: json!({}), body: "Content".into() })?;
atomic::write_atomic(&abs, raw.as_bytes())?;

// Step 4: Update SQLite store
wiki.writer().upsert_page(new_page).await?;

// Optional: Commit to git
if let Some(oid) = wiki.commit_all("Added Example page")? {
    println!("Created commit {}", oid);
}

Key Files and Their Roles

File	Purpose
crates/ai-memory-wiki/src/wiki.rs	Core Wiki struct; implements atomic write pipeline, mutation lock, and watcher coordination
crates/ai-memory-wiki/src/atomic.rs	Low-level helper for tempfile-write-then-rename atomicity
crates/ai-memory-wiki/src/watcher.rs	Git2-based filesystem watcher with tempfile filtering and re-index triggering

Summary

• Atomic writes via tempfile + rename guarantee the git2 watcher never sees partial page content
• Explicit tempfile filtering in the watcher prevents feedback loops from self-triggered events
• Shared mutation_lock serializes filesystem and store operations across writer and watcher
• Git2 integration observes raw filesystem events, not git commits, enabling immediate re-indexing
• SQLite synchronization occurs under the same lock as filesystem writes, ensuring consistency

Frequently Asked Questions

What prevents the git2 watcher from re-indexing its own temporary files?

The watcher implements ignores_own_atomic_tempfiles logic in watcher.rs (lines ~730-745), which checks if the changed path starts with .ai-memory-tmp. and skips processing if matched. This filter ensures only completed renames trigger re-indexing.

Why does ai-memory use atomic renames instead of direct writes?

Atomic renames on POSIX systems provide all-or-nothing visibility. The git2 watcher (and any other observer) sees either the previous file version or the complete new version, eliminating race conditions from partially written content.

How does the SQLite store stay synchronized with the filesystem?

Both writes occur while holding the exclusive mutation_lock. The upsert_page call to the SQLite writer actor executes in the same logical transaction as the atomic::write_atomic filesystem operation, ensuring crash consistency between disk and database.

Can admission webhooks block or modify a write before the git2 watcher sees it?

Yes. The admission chain executes after Markdown construction but before the atomic rename. Hooks can mutate frontmatter or return errors to abort the write. If aborted, no tempfile rename occurs and the git2 watcher remains unaware of the attempted change.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}