How to Export Palace Data Using the MemPalace Exporter Module

The MemPalace exporter module in mempalace/exporter.py provides the export_palace() function to convert ChromaDB palace storage into a browsable Markdown directory structure with built-in safety checks against symlink attacks and memory-efficient batch processing.

The MemPalace repository provides a dedicated exporter module that converts binary palace storage into structured Markdown documentation. To export palace data using the MemPalace exporter module, developers invoke the export_palace() function from mempalace/exporter.py, which streams drawer contents directly from the underlying ChromaDB collection without loading the entire dataset into memory. This approach generates a hierarchical file system representation while implementing security safeguards against path traversal and symlink attacks.

Understanding the Exporter Architecture

The core export logic resides in mempalace/exporter.py, which orchestrates data retrieval from mempalace/palace.py and safe file system operations. When you export palace data using the MemPalace exporter module, the system executes a seven-stage pipeline that maintains constant memory usage regardless of palace size.

The Export Pipeline

1. Collection Retrieval: Calls get_collection from mempalace/palace.py to establish a connection to the ChromaDB collection containing all drawers.
2. Directory Validation: Verifies the output path using _reject_symlink to prevent symbolic link attacks and creates the destination folder with restrictive permissions.
3. Paginated Streaming: Retrieves drawers in configurable batches (defaulting to 1000 per batch) to bound memory consumption.
4. Metadata Grouping: Organizes each batch by the wing and room metadata fields stored in drawer documents.
5. Markdown Generation: Creates an index.md file cataloging all wings alongside individual Markdown files for each room containing block-quoted drawer content and metadata tables.
6. File Handle Management: Tracks previously opened room files to determine whether to create (w) or append (a) content.
7. Statistics Collection: Aggregates drawer counts during iteration, returning a summary dictionary upon completion.

Programmatic Export Usage

Invoke the export functionality directly from Python code to integrate palace exports into larger workflows.

Basic Script Implementation

from mempalace.exporter import export_palace

# Path to the ChromaDB palace directory

palace_dir = "/home/user/.mempalace/palace"

# Destination for Markdown output

output_dir = "/tmp/palace_export"

# Execute export and retrieve statistics

stats = export_palace(palace_dir, output_dir)

print(f"Exported {stats['total_drawers']} drawers to {output_dir}")

Command Line Execution

While mempalace/cli.py provides a thin CLI wrapper, you can also execute the exporter directly via Python's -c flag or inline scripts:

python -c "
from mempalace.exporter import export_palace
export_palace(
    palace_path='/home/user/.mempalace/palace',
    output_dir='/tmp/palace_export'
)
"

Safety Mechanisms and File System Security

The MemPalace exporter module implements defensive programming patterns to handle untrusted input data safely.

Path Sanitization and Component Escaping

The _safe_path_component helper sanitizes wing and room names to ensure valid filesystem components, preventing directory traversal attacks through malicious metadata.

Symlink Attack Prevention

The exporter employs _reject_symlink to refuse writing into paths that are symbolic links, eliminating a class of time-of-check-time-of-use (TOCTOU) vulnerabilities. The _safe_open_for_write function further hardens file creation by utilizing O_NOFOLLOW flags where the operating system supports them, ensuring opened files are not symlink substitutions.

Output Structure and Generated Files

When you export palace data using the MemPalace exporter module, the system generates a hierarchical Markdown structure:

• Index File: A top-level index.md listing every wing with room counts and total drawer statistics.
• Wing Directories: Subdirectories for each wing with names escaped via _safe_path_component.
• Room Files: Individual <room>.md files within wing directories containing:
  • A Markdown heading identifying the room
  • Block-quoted drawer content
  • A metadata table documenting source, filed date, and added-by fields

This structure enables human browsing while preserving the relational organization of your memory palace.

Summary

• The export_palace() function in mempalace/exporter.py converts ChromaDB palace data into Markdown documentation.
• Export operations stream data in 1000-drawer batches to maintain constant memory usage for large palaces.
• Built-in safety helpers _safe_path_component, _reject_symlink, and _safe_open_for_write protect against path traversal and symlink attacks.
• The generated output includes an index.md file and hierarchical wing/room Markdown files with drawer metadata.
• The module integrates with mempalace/palace.py for database access and supports both programmatic and command-line invocation.

Frequently Asked Questions

What file format does the MemPalace exporter generate?

The exporter produces Markdown (.md) files. It creates an index.md at the root level containing wing and room statistics, plus individual room files organized within wing subdirectories. Each room file contains block-quoted drawer content alongside metadata tables showing source, filing date, and contributor information.

How does the exporter handle very large palace datasets?

Rather than loading the entire palace into RAM, the exporter streams drawers in paginated batches of 1000 by default. This pagination occurs within the main processing loop in mempalace/exporter.py, ensuring memory consumption remains bounded regardless of total drawer count.

Is the exporter safe to run against untrusted palace data?

Yes, the MemPalace exporter module implements multiple security layers. The _safe_path_component function sanitizes directory names derived from metadata, while _reject_symlink and _safe_open_for_write prevent symlink attacks and TOCTOU race conditions during file creation. These mechanisms allow safe execution even when palace metadata contains malicious path strings.

Where is the primary export logic implemented?

The main export functionality resides in mempalace/exporter.py, specifically within the export_palace() function. This module imports get_collection from mempalace/palace.py to access the underlying ChromaDB storage, while CLI access is available through mempalace/cli.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}