How the Magnitude Storage Package Persists Sessions and Configuration

The @magnitudedev/storage package provides a layered, effect‑based persistence layer that stores both session data and the global Magnitude configuration on the local file system using atomic file operations, schema validation, and concurrent‑safe locking.

The magnitudedev/magnitude repository implements a robust storage layer designed specifically for AI‑coding workflows. This TypeScript‑based system uses Effect‑TS to model side effects, errors, and concurrency while ensuring data integrity through schema‑based validation and atomic file operations.

Global Storage Root

All persistence begins with the global storage foundation defined in packages/storage/src/services/global-storage.ts. The GlobalStorageLive service creates a consistent root directory and supplies a GlobalStorageShape containing path helpers via makeGlobalStoragePaths.

Every storage component—whether for sessions or configuration—resolves its file locations relative to this root. This centralization ensures that the entire persistence layer can be relocated by configuring a single root path, making the system portable across different environments.

Session Persistence Architecture

Session data management is encapsulated in packages/storage/src/sessions/storage.ts through the makeSessionStorage() factory function. This constructs a SessionStorageShape that handles multiple data concerns through specialized file formats.

Session Metadata and Indexing

Each session maintains a JSON‑encoded metadata file (sessionMetaFile) storing the StoredSessionMeta schema. The storage layer uses readStructuredFile and writeStructuredFileAtomic (from structured-file.ts) to ensure schema validation during reads and atomic writes during updates. Errors trigger SchemaDecodeError or SchemaEncodeError for graceful recovery.

For fast lookups, the system maintains a per‑working‑directory index at cwdIndexFile. This JSON file maps a working directory to an ordered list of session IDs, allowing the CLI to quickly resolve recent sessions without scanning the entire storage tree.

Event Storage with JSON‑Lines

Session events are stored as timestamped entries in sessionEventsFile using the JSON‑Lines format. The io.recoverableJsonLines helper (defined in packages/storage/src/io/storage.ts) provides cursor‑based reading and append‑only writing, enabling incremental event streaming. If corruption occurs, the helper can repair the file by skipping malformed lines while preserving valid entries.

All file operations are guarded by io.withPathLock, which implements filesystem‑level locking to prevent race conditions during concurrent access from multiple processes.

Projection Snapshots and Addressed Entries

The storage layer supports complex state management through two additional mechanisms:

• Projection snapshots: Any computed projection state serializes to sessionProjectionSnapshotFile as a single JSON document, allowing the system to restore UI or analysis state without replaying the entire event stream.
• Addressed entries: A namespaced key/value store (sessionAddressedEntryFile) provides stable storage for specific values under schema‑enforced contracts, useful for caching derived data or user preferences.

Scratchpad Directories

Each session receives a dedicated scratch space via sessionScratchpad, which creates a fixed set of subdirectories defined in SCRATCHPAD_SUBDIRS. This isolation prevents temporary files from polluting the main storage area while maintaining session affinity for generated artifacts.

Configuration Persistence

Global Magnitude configuration is handled by makeConfigStorage() in packages/storage/src/config/storage.ts. This factory returns a ConfigStorageShape providing type‑safe access to the global config file.

The configuration system uses readRecoverableStructuredFile to load settings, defaulting to DEFAULT_CONFIG if the file is missing or malformed. Unlike strict session storage, the config layer attempts recovery from partially valid files, logging warnings rather than failing hard.

Updates are written atomically via writeStructuredFileAtomic, which preserves the original file until the new content is successfully flushed to disk. The API exposes high‑level operations including load, save, update, getContextLimitPolicy, and setContextLimitPolicy, all wrapped in Effect operations for consistent error handling.

Effect‑Based Concurrency and Error Handling

The entire storage layer is built on Effect‑TS principles. Services are provided through Effect layers (e.g., Effect.layerAll), allowing dependency injection of storage implementations. The system uses PubSub.sliding for broadcasting metadata changes across concurrent consumers, ensuring the UI or CLI remains synchronized with backend state changes.

This effect‑based approach guarantees that file system side effects, schema validations, and locking operations compose safely, preventing resource leaks and unhandled promise rejections common in imperative persistence code.

Summary

• The @magnitudedev/storage package creates a unified file‑system persistence layer under a configurable global root via GlobalStorageLive.
• Session storage handles metadata, event streams (JSON‑Lines), indexes, projections, and scratchpads using atomic operations and path‑based locking (io.withPathLock).
• Config storage provides recoverable, atomic read/write operations with sensible defaults and schema validation.
• All operations use Effect‑TS for type‑safe error handling and concurrency management, ensuring data integrity across concurrent processes.

Frequently Asked Questions

How does the storage package handle concurrent writes to the same session?

The package uses io.withPathLock (implemented in the I/O utilities) to acquire filesystem locks on specific file paths before writing. This prevents race conditions when multiple Magnitude processes attempt to append events or update metadata simultaneously, ensuring atomic updates even across separate CLI invocations.

What happens if a session metadata file becomes corrupted?

When reading sessionMetaFile, the storage layer uses readStructuredFile which validates the JSON against the StoredSessionMeta schema. If decoding fails, it returns a SchemaDecodeError that higher‑level services can catch to trigger recovery logic—typically falling back to default values or marking the session as recoverable rather than crashing the entire application.

Where does Magnitude store session data on disk?

All data resides under the global storage root defined by GlobalStorageLive in packages/storage/src/services/global-storage.ts. Within this root, sessions are organized into subdirectories by working directory (using the cwdIndexFile for mapping), with individual session folders containing the metadata, events, and scratchpad files.

Can I change the location where Magnitude stores its data?

Yes. Since all storage components depend on the GlobalStorageShape provided by GlobalStorageLive, you can configure the root directory path when constructing the Effect layer. All session storage (makeSessionStorage) and config storage (makeConfigStorage) operations derive their absolute paths from this root, making the entire persistence layer relocatable through a single configuration change.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}