How ai-memory Handles Atomic Markdown Writes: The tmp-rename-fsync Pattern Explained

ai-memory guarantees crash-safe Markdown file updates using a tmp-rename-fsync atomic write protocol with automatic rollback on database failures.

The ai-memory wiki engine stores every page as a Markdown file inside the <data_dir>/wiki/ directory tree. When you create or update a page, the system must ensure that no partially written or torn file ever appears on disk—even if the process crashes, the database transaction fails, or antivirus software briefly locks files on Windows. This article breaks down the exact mechanism implemented in the akitaonrails/ai-memory source code.

The Entry Point: Wiki::write_page

All page writes flow through Wiki::write_page at [wiki.rs lines 1803-1805](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs#L1803-L1805). This high-level method orchestrates sanitization, admission webhooks, atomic file replacement, database upserts, and optional embedding computation.

Here's the complete pipeline:

Step	Operation
Scrub & admission	Body sanitization, optional webhook mutation, front-matter canonicalization
Emit Markdown	In-memory Markdown struct serialized to UTF-8
Snapshot & atomic write	Existing file snapshot saved, new content written atomically
Store upsert	Page metadata upserted into SQLite; on failure, filesystem rolled back
Embedding	Vector embedding computed and stored (post-commit)
Post-write webhooks	Non-blocking hooks dispatched after durability confirmed

Core Primitive: atomic::write_atomic

The heart of the system lives in atomic::write_atomic at [atomic.rs lines 1-20](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/atomic.rs#L1-L20). This function implements the classic tmp-rename-fsync pattern for atomic file replacement:

// Example: atomically replace a page (used internally by write_page)
use ai_memory_wiki::atomic::write_atomic;
use std::path::Path;

let path = Path::new("/data_dir/wiki/001/002/notes.md");
let content = b"# Updated title\n\nNew body";

write_atomic(path, content)?;   // guarantees atomicity & fsync

The six-step protocol ensures durability:

1. Create temporary file in the target directory with prefix .ai-memory-tmp.
2. Write and sync data to the temporary file via tmp.as_file().sync_data()?
3. Atomic rename using tempfile::NamedTempFile::persist (with Windows retry logic)
4. Sync target file after rename via persisted.sync_data()?
5. Best-effort parent directory fsync via dir.sync_all()? to persist directory entry
6. Return inode/NTFS file index for the file watcher's self-ignore mechanism

Windows-Specific Retry Handling

On Windows, transient sharing violations from antivirus or search indexers can block the rename. The system uses persist_with_retry at [atomic.rs lines 20-40](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/atomic.rs#L20-L40), which retries the rename with 10ms delays before giving up.

Rollback Safety: Database-Filesystem Consistency

Atomic writes alone don't solve the two-phase commit problem: what if the file rename succeeds but the SQLite upsert fails? The ai-memory system solves this with snapshot-based rollback.

replace_file_with_rollback_snapshot

This helper at [wiki.rs lines 2303-2310](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs#L2303-L2310) captures the previous file state before overwriting:

// Example: manually rolling back a failed write (illustrates the helper)
use ai_memory_wiki::wiki::{replace_file_with_rollback_snapshot, rollback_or_inconsistent};

let installed = replace_file_with_rollback_snapshot(&path, bytes)?;
if let Err(db_err) = writer.upsert_page(...).await {
    // Restore the previous version so DB and FS stay consistent
    rollback_or_inconsistent(&[installed], &db_err)?;
}

The function returns an InstalledFile struct recording:

• The old file bytes (or None if file didn't exist)
• The target path

rollback_installed_files

On store-write failure, rollback_installed_files at [wiki.rs lines 2312-2325](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs#L2312-L2325) executes:

• For existing files: Write back the saved snapshot bytes
• For new files: Remove the newly-created file

This ensures the on-disk state always matches the database state, even after crashes.

Complete write_page Usage

Here's how client code triggers the full atomic workflow:

// Example: how write_page performs a safe replace with rollback
use ai_memory_wiki::wiki::Wiki;
use ai_memory_core::WritePageRequest;

let req = WritePageRequest {
    workspace_id,
    project_id,
    path: PagePath::new("notes/todo.md")?,
    frontmatter: serde_json::json!({ "tags": ["todo"] }),
    body: "Fix the atomic write implementation".into(),
    ..Default::default()
};

let page_id = wiki.write_page(req).await?;   // atomic write + DB upsert

File Watcher Integration: Self-Ignoring Writes

The atomic write system returns the inode (or NTFS file index) of the new file at [atomic.rs lines 95-140](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/atomic.rs#L95-L140). The file watcher uses this to distinguish its own writes from external modifications, preventing infinite update loops.

Key Source Files

File	Purpose
[crates/ai-memory-wiki/src/atomic.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/atomic.rs)	write_atomic, persist_with_retry, inode extraction—low-level atomic primitives
[crates/ai-memory-wiki/src/wiki.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/wiki.rs)	write_page, replace_file_with_rollback_snapshot, rollback orchestration
[crates/ai-memory-wiki/src/markdown.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/markdown.rs)	Markdown serialization used before atomic write
[crates/ai-memory-wiki/src/admission.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/src/admission.rs)	Pre-write webhook chain for mutation/rejection
[crates/ai-memory-wiki/tests/suite/mod.rs](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-wiki/tests/suite/mod.rs)	Test coverage for atomic writes, rollback, Windows retry logic

Summary

• Atomic markdown writes in ai-memory use the proven tmp-rename-fsync pattern with six durability steps
• Windows retry logic handles transient antivirus locks via persist_with_retry
• Snapshot rollback ensures database-filesystem consistency when SQLite upserts fail
• Inode tracking lets the file watcher ignore its own atomic writes
• Admission webhooks run before the atomic commit, allowing safe content mutation

Frequently Asked Questions

What happens if the process crashes during an atomic write?

If the crash occurs before the rename, the temporary file is abandoned and the original file remains intact. If the crash occurs after the rename but before the database commit, the next operation will detect the inconsistency and trigger rollback_or_inconsistent to restore the previous file state.

Does ai-memory fsync on every write, and does this hurt performance?

Yes, ai-memory calls sync_data() on both the temporary file and the persisted file, plus sync_all() on the parent directory. These are blocking operations that ensure crash safety. The design prioritizes durability over raw throughput—appropriate for a knowledge base where data integrity matters more than log-style append performance.

Why does the system need to track inodes after atomic writes?

The file watcher monitors the <data_dir>/wiki/ tree for external changes. Without inode tracking, the watcher would detect its own atomic renames as external modifications and potentially re-process or re-index the file. By recording the inode of files it writes, the watcher can filter self-generated events.

Can the atomic write fail on Windows even with retry logic?

In extreme cases—antivirus software with aggressive file locking, or very slow I/O—the 10ms × N retry loop in persist_with_retry may exhaust its attempts. The operation returns an error, which bubbles up to write_page and triggers the standard rollback path. No partial file state is ever committed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}