How ai-memory Implements Forgot-Sweep TTL Deletion of Expired Wiki Pages

ai-memory uses a two-phase expiration system: query-time filtering hides expired pages immediately, while a background "forgot-sweep" task permanently deletes them from disk and database.

The akit ails/ai-memory repository implements automatic time-based expiration for wiki pages through a coordinated mechanism spanning front-matter parsing, SQLite storage, read filtering, and a periodic cleanup task. This article explains exactly how the forgot-sweep TTL deletion works under the hood.

Parsing TTL from Front-Matter

When a markdown file is written to the wiki, the parse_expires_at function in crates/ai-memory-wiki/src/wiki.rs extracts the optional expires_at value from YAML front-matter. It accepts RFC-3339 timestamps or simple YYYY-MM-DD dates and converts them to a jiff::Timestamp (source lines 1927–1940).

---
title: "Temporary Note"
expires_at: 2025-12-31T23:59:59Z   # RFC-3339 timestamp

---
Content that should disappear after the TTL.

This timestamp is persisted to SQLite via upsert operations in crates/ai-memory-store/src/ops.rs, specifically in the pages.expires_at column.

Preventing Expired Page Reads

Every query that reads wiki pages automatically excludes expired rows. In crates/ai-memory-store/src/reader.rs, the query builder injects this guard clause (source line 51):

AND (pages.expires_at IS NULL OR pages.expires_at > ?now)

The ?now placeholder receives the current UTC microsecond timestamp. This ensures expired pages are immediately invisible to all normal operations, even before the sweep runs.

The Forgot-Sweep Background Task

The actual deletion happens in run_forgot_sweep, an async task defined in crates/ai-memory-wiki/src/wiki.rs. This task uses tokio::time::interval with a default 1-hour period to scan for stale entries.

For each expired row found, the sweep:

• Calls Wiki::hard_delete_page to remove the markdown file from the filesystem
• Executes DELETE FROM pages … to drop the database record

This hard-delete is unconditional—if the file was already removed manually, the database row is still deleted, guaranteeing complete cleanup.

Task Initialization and Thread Safety

The sweep task starts when the wiki subsystem initializes in crates/ai-memory-wiki/src/lib.rs. Because ai-memory uses a single-writer SQLite actor pattern, the sweep runs on the same thread that handles all database writes, avoiding connection contention.

Manual Sweep Invocation

For testing or administrative use, the Wiki type exposes run_forgot_sweep_once:

use ai_memory_wiki::Wiki;

// Assume `wiki` is an initialized `Wiki` instance.
wiki.run_forgot_sweep_once().await?;

This performs one immediate scan and deletes all rows where expires_at ≤ now.

Summary

• parse_expires_at in wiki.rs converts front-matter dates to jiff::Timestamp values
• reader.rs filters expired rows from every SELECT query automatically
• run_forgot_sweep in wiki.rs runs hourly to hard-delete expired pages from disk and SQLite
• The sweep respects the single-writer SQLite actor invariant by running on the writer thread
• run_forgot_sweep_once allows manual or test-triggered cleanup

Frequently Asked Questions

How does ai-memory prevent reading expired wiki pages before the sweep runs?

The query layer in crates/ai-memory-store/src/reader.rs automatically adds AND (pages.expires_at IS NULL OR pages.expires_at > ?now) to every SELECT statement. This filter uses the current UTC microsecond timestamp, making expired rows invisible immediately upon expiration without waiting for the background sweep.

What timestamp formats does ai-memory accept for expires_at?

The parse_expires_at function accepts RFC-3339 timestamps (e.g., 2025-12-31T23:59:59Z) or simple YYYY-MM-DD dates. Both are normalized to jiff::Timestamp for storage and comparison.

Why does the forgot-sweep use hard deletion instead of soft deletion?

The sweep calls Wiki::hard_delete_page and executes DELETE FROM pages … unconditionally. This guarantees that expired content disappears completely from both filesystem and database, aligning with the "forgot" semantics—data with an elapsed TTL should not be recoverable through normal means.

Can I trigger the forgot-sweep manually or in tests?

Yes. The Wiki type provides run_forgot_sweep_once().await, which performs a single immediate scan and deletes all expired rows. This is designed for unit tests and administrative scenarios where you need synchronous cleanup rather than waiting for the hourly interval.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}