# How Finestore Manages Long-Term Artifact Storage and Lazy Artifact Retrieval in Marin

> Discover how Finestore ensures durable artifact storage and lazy retrieval in Marin using immutable Parquet archives and efficient ReadViews. Learn about its advanced persistence and I/O deferral.

- Repository: [The Marin Project/marin](https://github.com/marin-community/marin)
- Tags: deep-dive
- Published: 2026-08-28

---

**Finestore implements durable artifact persistence through immutable Parquet archives with append-only transactional writes, while enabling efficient lazy retrieval via snapshot-isolated ReadViews and an in-process PersistentKvCache that defers remote I/O.**

Finestore is the storage engine within the [marin-community/marin](https://github.com/marin-community/marin) repository designed for machine learning workflows requiring both durable versioning and high-performance access. The system separates write-heavy ingestion from read-heavy retrieval by combining an **append-only transactional writer** with **snapshot-based readers** and a **tiered caching layer**, ensuring artifacts remain immutable and compact over the long term without sacrificing query performance.

## Long-Term Artifact Storage Architecture

Finestore persists artifacts—including binary blobs, tables, and schema—within a **FineStore archive**, a directory containing an immutable manifest and a collection of Parquet shards. This design prioritizes durability and immutability while allowing efficient compaction of historical data.

### Archive Initialization and Structure

When a writer initializes storage via `DataStore.open(root, …)`, the constructor builds a `FineStoreLayout` for the specified root path and calls `initialize_archive(self._layout)` to prepare the archive directory. According to the implementation in [`lib/finestore/src/finestore/store.py`](https://github.com/marin-community/marin/blob/main/lib/finestore/src/finestore/store.py), this establishes the foundation for all subsequent write operations by creating the necessary metadata structures and shard directories.

### Transactional Writes and Buffering

Artifacts are written through `DataStore.write_object(name, data, metadata)`, which creates a **blob descriptor** and optional chunked parts, appending them to the internal `blobs` table. Writes are buffered in-memory per-table to optimize throughput. When the buffer exceeds `max_buffer_bytes` or an explicit `flush()` is invoked, the buffered rows convert into a **ShardWriter** and commit as a **level-zero shard**.

The commit sequence operates through the `CommitCoordinator`, recording metadata in the manifest to ensure the archive remains readable and consistent. This append-only approach prevents in-place modifications, preserving the history of artifact versions within [`lib/finestore/src/finestore/store.py`](https://github.com/marin-community/marin/blob/main/lib/finestore/src/finestore/store.py).

### Background Maintenance and Compaction

Long-term storage efficiency relies on a background maintenance thread (`self._thread`) that regularly executes `self.maintain()`. This process compacts over-full shards and optionally seals the archive, transforming multiple small shards into fewer, larger files to improve read performance. The maintenance routine defined in [`lib/finestore/src/finestore/store.py`](https://github.com/marin-community/marin/blob/main/lib/finestore/src/finestore/store.py) ensures that storage remains compact and immutable over extended periods without blocking active write operations.

## Lazy Artifact Retrieval Mechanisms

Retrieval operations in Finestore avoid eager materialization of entire archives. Instead, the system uses snapshot isolation and streaming iterators to access only the necessary data segments.

### Snapshot-Based ReadViews

Marin reads artifacts through the read-only `ReadView` class, which **pins a snapshot** of the manifest at construction time. According to [`lib/finestore/src/finestore/reader.py`](https://github.com/marin-community/marin/blob/main/lib/finestore/src/finestore/reader.py), this snapshot isolation guarantees consistent reads even as concurrent writes append new shards to the archive.

The `ReadView.point(table, **keys)` and `ReadView.iter_rows(...)` methods resolve the latest committed rows without loading the complete archive. These iterators stream rows from relevant shards while applying push-down filters and deduplication on the fly, minimizing memory overhead for large datasets.

### Streaming Blob Access

For named binary objects, `ReadView.read_blob(name)` opens a forward-only `_BlobReader` stream. This implementation in [`lib/finestore/src/finestore/reader.py`](https://github.com/marin-community/marin/blob/main/lib/finestore/src/finestore/reader.py) yields inline data or ordered chunked parts only when the consumer reads them, enabling lazy loading of large model checkpoints or datasets without holding the entire artifact in memory.

### PersistentKvCache Memory Tier

The `PersistentKvCache` provides an **in-process memory tier** that bridges hot in-memory access with cold archive storage. As implemented in [`lib/finestore/src/finestore/cache.py`](https://github.com/marin-community/marin/blob/main/lib/finestore/src/finestore/cache.py), the cache operates with distinct read and write paths:

**Lazy Reads:** The `load(key)` method first checks the local dictionary; on a cache miss, it pins a `ReadView` and reads the blob from the archive, subsequently storing the result in memory for future fast hits.

**Asynchronous Writes:** The `store(key, value)` method writes immediately to the local memory dict. When configured for remote persistence (via `marin_temp_bucket`), the operation queues the write via `_queue_remote_write`. A daemon thread (`_drain_remote_writes`) drains this queue in the background, batching writes into an **unbounded transaction** and committing them to the FineStore archive in a single flush. This ensures the archive remains eventually consistent while the application experiences instant availability from the memory cache.

## Implementation Examples

The following examples demonstrate the interaction between long-term storage and lazy retrieval:

```python

# Writing artifacts to long-term storage

from finestore.store import DataStore

store = DataStore.open("/mnt/runs/run-001")  # Initialize archive

uri = store.write_object("checkpoint.pt", model_bytes)  # Create blob descriptor

store.flush()  # Commit level-zero shard via CommitCoordinator

```

```python

# Lazy retrieval without full archive scan

from finestore.reader import ReadView

view = ReadView("/mnt/runs/run-001")  # Pin manifest snapshot

data = view.read_blob("checkpoint.pt")  # Stream bytes on demand

```

```python

# Using PersistentKvCache for lazy reads and async writes

from finestore.cache import PersistentKvCache

# Writer process

writer_cache = PersistentKvCache.for_prefix("experiment-1", is_writer=lambda: True)
writer_cache.store("metrics", metrics_blob)  # Local memory + queued remote write

# Reader process

reader_cache = PersistentKvCache.for_prefix("experiment-1")
result = reader_cache.load("metrics")  # Memory hit or lazy archive read

reader_cache.close()  # Drain pending writes

```

## Summary

- **Finestore** stores artifacts in immutable Parquet archives with append-only transactional writes, ensuring durable, versioned storage without in-place modifications.
- The `DataStore` class manages buffered writes, shard creation, and background compaction via `CommitCoordinator` and maintenance threads defined in [`lib/finestore/src/finestore/store.py`](https://github.com/marin-community/marin/blob/main/lib/finestore/src/finestore/store.py).
- **Lazy retrieval** relies on `ReadView` snapshots that pin manifests at construction time, streaming only required rows or blob chunks via iterators in [`lib/finestore/src/finestore/reader.py`](https://github.com/marin-community/marin/blob/main/lib/finestore/src/finestore/reader.py).
- The `PersistentKvCache` provides tiered access with in-process memory caching, lazy archive reads on misses, and asynchronous background writes for remote persistence in [`lib/finestore/src/finestore/cache.py`](https://github.com/marin-community/marin/blob/main/lib/finestore/src/finestore/cache.py).

## Frequently Asked Questions

### How does Finestore ensure data durability during writes?

Finestore ensures durability through an append-only transaction model. When `DataStore.flush()` is called, buffered rows convert to a `ShardWriter` and commit as a level-zero shard through the `CommitCoordinator`, which atomically records the operation in the immutable manifest. This guarantees that once a write completes, the data remains permanently stored in the Parquet shards regardless of subsequent operations.

### What triggers background maintenance in Finestore?

A dedicated background thread (`self._thread`) runs `self.maintain()` at regular intervals to compact over-full shards and optionally seal the archive. This maintenance prevents the accumulation of small fragmentary shards by merging them into larger, more efficient files, ensuring long-term storage remains optimized for both space and read performance without blocking active writes.

### How does ReadView provide consistent reads during concurrent writes?

`ReadView` implements snapshot isolation by pinning the manifest at construction time. This creates a consistent point-in-time view of the archive, allowing readers to call `point()`, `iter_rows()`, or `read_blob()` without seeing partial or concurrent modifications. The iterator streams from shards referenced in the pinned manifest while applying filters, ensuring deterministic results even as new shards are appended.

### Can PersistentKvCache operate without remote storage?

Yes, `PersistentKvCache` functions effectively as a local in-memory cache even without remote storage configuration. When `marin_temp_bucket` is not specified, `store()` operations populate only the local dictionary, and `load()` operations serve memory hits or read directly from the local FineStore archive. The background write queue and `_drain_remote_writes` thread activate only when remote persistence is configured, making the cache adaptable to both single-node and distributed deployments.