How the Q00/ouroboros EventStore Reconstructs Lineage Across Restarts: Event Sourcing Deep Dive

The Q00/ouroboros EventStore reconstructs lineage across restarts by replaying persisted "lineage" aggregate events from SQLite and folding them through a deterministic LineageProjector to rebuild the immutable OntologyLineage state.

When the Ouroboros process stops, the evolutionary history of an ontology—its lineage—is lost from memory. To ensure continuity, the Q00/ouroboros repository implements an event-sourced persistence model where the EventStore serves as the single source of truth. This article explains exactly how the EventStore reconstructs lineage across restarts by replaying events from src/ouroboros/persistence/schema.py and projecting them through LineageProjector.

The Event Sourcing Foundation for Lineage Persistence

Unified Event Storage in SQLite

All domain events, including lineage mutations, share a single append-only table defined in src/ouroboros/persistence/schema.py:

# src/ouroboros/persistence/schema.py

events_table = Table(
    "events",
    metadata,
    Column("id", String(36), primary_key=True),
    Column("aggregate_type", String(100), nullable=False),
    Column("aggregate_id", String(36), nullable=False),
    Column("event_type", String(200), nullable=False),
    Column("payload", JSON, nullable=False),
    Column("timestamp", DateTime(timezone=True), nullable=False, …),
)

Lineage-specific events are distinguished by aggregate_type = "lineage" and aggregate_id = <lineage-id>, with event_type values such as lineage.created, lineage.generation.started, lineage.rewound, and terminal states like lineage.converged.

The store writes them with EventStore.append and append_batch (see lines 71‑92 and 100‑112 of event_store.py).

Appending Lineage Events

Before reconstruction can occur, events must be persisted using the BaseEvent class from src/ouroboros/events/base.py:

event = BaseEvent(
    id=str(uuid4()),
    aggregate_type="lineage",
    aggregate_id="lin_123abc",
    type="lineage.created",
    data={"goal": "Create a personal finance ontology"},
    timestamp=datetime.now(UTC),
)
await store.append(event)

Replaying Lineage Events on Startup

The replay_lineage Method

When Ouroboros restarts, the EventStore reconstructs state by replaying only the events belonging to a specific lineage aggregate. The entry point is replay_lineage in src/ouroboros/persistence/event_store.py:

# src/ouroboros/persistence/event_store.py

async def replay_lineage(self, lineage_id: str) -> list[BaseEvent]:
    """Replay all events for a lineage aggregate."""
    return await self.replay("lineage", lineage_id)   # ← lines 414‑428

This delegates to the generic replay method, which queries the SQLite events table filtering on aggregate_type and aggregate_id.

Deterministic Event Ordering

To guarantee that reconstruction yields the exact same state across every restart, the replay method orders results by timestamp, id (lines 66‑73 of event_store.py):

-- Conceptual SQL generated by the replay method
SELECT * FROM events 
WHERE aggregate_type = ? AND aggregate_id = ? 
ORDER BY timestamp, id

This deterministic ordering ensures that the LineageProjector processes events in the precise sequence they were originally emitted, preserving causal relationships between generations, rewinds, and terminal states.

Projecting Events into Domain State

The LineageProjector.fold Logic

Raw events must be transformed into the domain model. The LineageProjector in src/ouroboros/evolution/projector.py implements a left-fold (reduce) operation that iterates through the event list and builds an OntologyLineage instance:

# src/ouroboros/evolution/projector.py

class LineageProjector:
    def project(self, events: list[BaseEvent]) -> OntologyLineage | None:
        if not events:
            return None                                   # lines 43‑44

        lineage: OntologyLineage | None = None
        generations: dict[int, GenerationRecord] = {}
        rewind_history: list[RewindRecord] = []

        for event in events:                              # lines 50‑66

            if event.type == "lineage.created":
                lineage = OntologyLineage(
                    lineage_id=event.aggregate_id,
                    goal=event.data.get("goal", ""),
                    created_at=event.timestamp,
                )
            elif event.type == "lineage.generation.started":
                # build a pending GenerationRecord

                pass
            elif event.type == "lineage.generation.completed":
                # build a completed GenerationRecord

                pass
            elif event.type == "lineage.generation.failed":
                # mark failure

                pass
            elif event.type == "lineage.rewound":
                # record rewind & truncate generations

                pass
            elif event.type in {"lineage.converged",
                                "lineage.exhausted",
                                "lineage.stagnated"}:
                lineage = lineage.with_status(...)
        # after the loop build the final immutable model

        sorted_records = tuple(generations[k] for k in sorted(generations))
        return lineage.model_copy(
            update={"generations": sorted_records,
                    "rewind_history": tuple(rewind_history)},
        )

The projector maintains three accumulators: the root lineage object, a dictionary of generations, and a list of rewind_history. It pattern-matches on event.type to apply the appropriate state transition, ensuring that rewinds properly truncate generation history and terminal events update the lineage status.

Immutability and the OntologyLineage Model

The domain model itself is defined in src/ouroboros/core/lineage.py as an immutable data structure. The projector never mutates the stored events; instead, it creates a fresh OntologyLineage instance that represents the cumulative effect of the entire event stream. This aligns with Event Sourcing principles where the event log is the single source of truth and domain objects are ephemeral projections.

Complete Startup Flow Example

The following snippet demonstrates the exact sequence used by Ouroboros to reconstruct lineage after a restart:

from ouroboros.persistence.event_store import EventStore
from ouroboros.evolution.projector import LineageProjector

async def load_lineage(lineage_id: str) -> OntologyLineage | None:
    store = EventStore()                     # default ~/.ouroboros/ouroboros.db

    await store.initialize()
    events = await store.replay_lineage(lineage_id)   # ← fetch persisted events

    projector = LineageProjector()
    lineage = projector.project(events)               # ← reconstruct state

    await store.close()
    return lineage

If the process crashes or is stopped, executing load_lineage on the next run will re‑play the exact same event sequence, guaranteeing that the lineage is restored faithfully without data loss.

Summary

• Event Store as Source of Truth: The events table in src/ouroboros/persistence/schema.py stores all lineage mutations as immutable records with aggregate_type="lineage".
• Deterministic Replay: The EventStore.replay_lineage method in src/ouroboros/persistence/event_store.py queries events ordered by timestamp, id to ensure consistent reconstruction order.
• Stateless Projection: The LineageProjector.project method in src/ouroboros/evolution/projector.py folds the event stream into a fresh OntologyLineage instance without database side effects.
• Crash Safety: Because reconstruction happens entirely from the persisted event log, stopping or crashing the process never corrupts lineage state; the next startup simply replays the log to reach the same deterministic state.

Frequently Asked Questions

What happens if the SQLite database is corrupted or deleted?

If the SQLite database file (default ~/.ouroboros/ouroboros.db) is lost or corrupted, the event log is destroyed and lineage reconstruction becomes impossible. Ouroboros treats the event store as the sole source of truth, so proper backup strategies for the SQLite file are essential for production deployments.

How does the EventStore handle concurrent writes to the same lineage?

The EventStore.append and append_batch methods in src/ouroboros/persistence/event_store.py utilize SQLite's transaction isolation. Because SQLite handles file-level locking and the event store appends are atomic operations, concurrent processes writing to the same lineage aggregate will be serialized by the database engine, preventing event loss or ordering corruption.

Can the lineage reconstruction process be optimized for large event logs?

Currently, the replay_lineage method loads all events for a given lineage into memory before projection. For lineages with thousands of generations, you could implement snapshotting by persisting periodic checkpoints of the OntologyLineage state and modifying the projector to load only events after the latest snapshot. However, the current implementation in src/ouroboros/evolution/projector.py performs a full fold from the beginning of the event stream.

What is the difference between lineage.rewound and lineage.generation.failed events?

A lineage.rewound event represents an intentional rollback to a previous generation, triggering the LineageProjector to truncate the generations dictionary and record a RewindRecord. In contrast, a lineage.generation.failed event marks a specific generation attempt as unsuccessful without altering the lineage history; the failed generation remains in the record as a terminal state for that specific attempt, allowing the system to retry or abandon that path without discarding prior successful work.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}