# How the Magnitude Storage Package Persists Sessions and Configuration

> Discover how the magnitudedev storage package persists sessions and config using atomic file operations, schema validation, and concurrent safe locking for robust local file system storage.

- Repository: [Magnitude/magnitude](https://github.com/magnitudedev/magnitude)
- Tags: internals
- Published: 2026-09-08

---

**The `@magnitudedev/storage` package provides a layered, effect‑based persistence layer that stores both session data and the global Magnitude configuration on the local file system using atomic file operations, schema validation, and concurrent‑safe locking.**

The `magnitudedev/magnitude` repository implements a robust storage layer designed specifically for AI‑coding workflows. This TypeScript‑based system uses **Effect‑TS** to model side effects, errors, and concurrency while ensuring data integrity through schema‑based validation and atomic file operations.

## Global Storage Root

All persistence begins with the global storage foundation defined in [`packages/storage/src/services/global-storage.ts`](https://github.com/magnitudedev/magnitude/blob/main/packages/storage/src/services/global-storage.ts). The `GlobalStorageLive` service creates a consistent root directory and supplies a `GlobalStorageShape` containing path helpers via `makeGlobalStoragePaths`.

Every storage component—whether for sessions or configuration—resolves its file locations relative to this root. This centralization ensures that the entire persistence layer can be relocated by configuring a single root path, making the system portable across different environments.

## Session Persistence Architecture

Session data management is encapsulated in [`packages/storage/src/sessions/storage.ts`](https://github.com/magnitudedev/magnitude/blob/main/packages/storage/src/sessions/storage.ts) through the `makeSessionStorage()` factory function. This constructs a `SessionStorageShape` that handles multiple data concerns through specialized file formats.

### Session Metadata and Indexing

Each session maintains a JSON‑encoded metadata file (`sessionMetaFile`) storing the `StoredSessionMeta` schema. The storage layer uses `readStructuredFile` and `writeStructuredFileAtomic` (from [`structured-file.ts`](https://github.com/magnitudedev/magnitude/blob/main/structured-file.ts)) to ensure schema validation during reads and atomic writes during updates. Errors trigger `SchemaDecodeError` or `SchemaEncodeError` for graceful recovery.

For fast lookups, the system maintains a per‑working‑directory index at `cwdIndexFile`. This JSON file maps a working directory to an ordered list of session IDs, allowing the CLI to quickly resolve recent sessions without scanning the entire storage tree.

### Event Storage with JSON‑Lines

Session events are stored as timestamped entries in `sessionEventsFile` using the **JSON‑Lines** format. The `io.recoverableJsonLines` helper (defined in [`packages/storage/src/io/storage.ts`](https://github.com/magnitudedev/magnitude/blob/main/packages/storage/src/io/storage.ts)) provides cursor‑based reading and append‑only writing, enabling incremental event streaming. If corruption occurs, the helper can repair the file by skipping malformed lines while preserving valid entries.

All file operations are guarded by `io.withPathLock`, which implements filesystem‑level locking to prevent race conditions during concurrent access from multiple processes.

### Projection Snapshots and Addressed Entries

The storage layer supports complex state management through two additional mechanisms:

- **Projection snapshots**: Any computed projection state serializes to `sessionProjectionSnapshotFile` as a single JSON document, allowing the system to restore UI or analysis state without replaying the entire event stream.
- **Addressed entries**: A namespaced key/value store (`sessionAddressedEntryFile`) provides stable storage for specific values under schema‑enforced contracts, useful for caching derived data or user preferences.

### Scratchpad Directories

Each session receives a dedicated scratch space via `sessionScratchpad`, which creates a fixed set of subdirectories defined in `SCRATCHPAD_SUBDIRS`. This isolation prevents temporary files from polluting the main storage area while maintaining session affinity for generated artifacts.

## Configuration Persistence

Global Magnitude configuration is handled by `makeConfigStorage()` in [`packages/storage/src/config/storage.ts`](https://github.com/magnitudedev/magnitude/blob/main/packages/storage/src/config/storage.ts). This factory returns a `ConfigStorageShape` providing type‑safe access to the global config file.

The configuration system uses `readRecoverableStructuredFile` to load settings, defaulting to `DEFAULT_CONFIG` if the file is missing or malformed. Unlike strict session storage, the config layer attempts recovery from partially valid files, logging warnings rather than failing hard.

Updates are written atomically via `writeStructuredFileAtomic`, which preserves the original file until the new content is successfully flushed to disk. The API exposes high‑level operations including `load`, `save`, `update`, `getContextLimitPolicy`, and `setContextLimitPolicy`, all wrapped in Effect operations for consistent error handling.

## Effect‑Based Concurrency and Error Handling

The entire storage layer is built on **Effect‑TS** principles. Services are provided through Effect layers (e.g., `Effect.layerAll`), allowing dependency injection of storage implementations. The system uses `PubSub.sliding` for broadcasting metadata changes across concurrent consumers, ensuring the UI or CLI remains synchronized with backend state changes.

This effect‑based approach guarantees that file system side effects, schema validations, and locking operations compose safely, preventing resource leaks and unhandled promise rejections common in imperative persistence code.

## Summary

- The `@magnitudedev/storage` package creates a unified file‑system persistence layer under a configurable global root via `GlobalStorageLive`.
- **Session storage** handles metadata, event streams (JSON‑Lines), indexes, projections, and scratchpads using atomic operations and path‑based locking (`io.withPathLock`).
- **Config storage** provides recoverable, atomic read/write operations with sensible defaults and schema validation.
- All operations use Effect‑TS for type‑safe error handling and concurrency management, ensuring data integrity across concurrent processes.

## Frequently Asked Questions

### How does the storage package handle concurrent writes to the same session?

The package uses `io.withPathLock` (implemented in the I/O utilities) to acquire filesystem locks on specific file paths before writing. This prevents race conditions when multiple Magnitude processes attempt to append events or update metadata simultaneously, ensuring atomic updates even across separate CLI invocations.

### What happens if a session metadata file becomes corrupted?

When reading `sessionMetaFile`, the storage layer uses `readStructuredFile` which validates the JSON against the `StoredSessionMeta` schema. If decoding fails, it returns a `SchemaDecodeError` that higher‑level services can catch to trigger recovery logic—typically falling back to default values or marking the session as recoverable rather than crashing the entire application.

### Where does Magnitude store session data on disk?

All data resides under the global storage root defined by `GlobalStorageLive` in [`packages/storage/src/services/global-storage.ts`](https://github.com/magnitudedev/magnitude/blob/main/packages/storage/src/services/global-storage.ts). Within this root, sessions are organized into subdirectories by working directory (using the `cwdIndexFile` for mapping), with individual session folders containing the metadata, events, and scratchpad files.

### Can I change the location where Magnitude stores its data?

Yes. Since all storage components depend on the `GlobalStorageShape` provided by `GlobalStorageLive`, you can configure the root directory path when constructing the Effect layer. All session storage (`makeSessionStorage`) and config storage (`makeConfigStorage`) operations derive their absolute paths from this root, making the entire persistence layer relocatable through a single configuration change.