How Semantica Ensures Thread-Safe In‑Memory Property Graphs: A Deep Dive into the Codebase

Semantica uses a multi-layered threading strategy combining explicit locks, thread-local storage, re-entrant locks, and driver-level session isolation to guarantee safe concurrent access to its in‑memory property graphs.

The semantica-agi/semantica repository implements a flexible graph storage architecture that supports both external databases (Neo4j, FalkorDB, Amazon Neptune, Apache AGE) and lightweight in‑memory backends. When operating in‑memory—particularly with SQLite-backed storage—thread safety becomes critical to prevent race conditions, data corruption, and driver errors. This article examines the exact mechanisms Semantica employs, with direct references to source file paths and implementation details.

Core Thread-Safety Mechanisms

Semantica's threading model rests on five complementary techniques. Each addresses a specific concurrency risk in the property graph pipeline.

Explicit threading.Lock for SQLite Operations

The primary defense against concurrent mutation lives in semantica/vector_store/sqlite_vec_store.py. This module implements an in‑memory vector store that doubles as a property graph backend using SQLite.

# From semantica/vector_store/sqlite_vec_store.py

import threading

class SQLiteVecStore:
    def __init__(self, ...):
        self._lock = threading.Lock()  # Line 139-140: exclusive lock creation

        ...
        self._conn = sqlite3.connect(
            ...,
            check_same_thread=False  # Line 160-172: enables cross-thread sharing

        )
    
    def add(self, ...):
        with self._lock:               # Exclusive section begins

            self._conn.execute(sql, params)  # Safe write operation

The check_same_thread=False flag is essential here: it allows a single SQLite connection to be shared across threads, but only because every write operation is serialized by self._lock. Without this lock, SQLite would throw ProgrammingError: SQLite objects created in a thread can only be used in that thread.

Thread-Local Storage for Per-Thread Buffers

Not all shared state requires locking. For transient data like provenance entries, Semantica uses threading.local to give each thread its own isolated workspace:

# From semantica/provenance/storage.py

import threading

class ProvenanceStorage:
    def __init__(self):
        self._local = threading.local()  # Line 216-219: thread-local container

    
    def add_entry(self, entry):
        if not hasattr(self._local, "buffer"):
            self._local.buffer = []      # Each thread gets independent list

        self._local.buffer.append(entry) # Zero contention with other threads

This pattern eliminates lock overhead for high-frequency append operations while maintaining clean separation between concurrent pipeline stages.

Re-Entrant Locks for Recursive Resource Allocation

The pipeline scheduler in semantica/pipeline/resource_scheduler.py faces a more complex scenario: resource allocation may trigger nested callbacks that re-enter the same critical section. For this, Semantica uses threading.RLock:

# From semantica/pipeline/resource_scheduler.py

import threading

class ResourceScheduler:
    def __init__(self):
        self.lock = threading.RLock()    # Line 112-113: re-entrant lock

    
    def schedule(self, task):
        with self.lock:
            self.allocate_resources(task)  # Safe even if allocate_resources

                                          # internally calls schedule() again

RLock allows the same thread to acquire the lock multiple times without deadlocking—critical for recursive scheduling algorithms.

Driver-Level Session Isolation for External Stores

While SQLite handles in‑memory graphs, Semantica also integrates with external graph databases. For these, thread safety is achieved at the driver level by never sharing sessions across threads:

Store Implementation	Session Pattern
semantica/graph_store/neo4j_store.py	New Session per operation, immediately closed
semantica/graph_store/falkordb_store.py	Fresh client connection per thread
Neptune, AGE implementations	Thread-scoped connection pooling

In neo4j_store.py, each graph operation follows this pattern:

# Conceptual pattern from neo4j_store.py implementations

def execute_query(self, cypher: str, parameters: dict):
    with self.driver.session() as session:  # Session created and destroyed locally

        result = session.run(cypher, parameters)
        return result.data()

Short-lived sessions prevent thread interference without requiring explicit locks in Python code—the Neo4j driver handles concurrency internally.

SQLite Connection Sharing with Serialized Access

The complete threading contract for SQLite-backed storage appears in sqlite_vec_store.py. The combination of:

• check_same_thread=False on connection creation
• threading.Lock() around all execute/commit operations
• Context manager protocol (with self._lock:)

...yields a locked, shareable connection that safely multiplexes access across any number of threads.

Where Thread Safety Matters in the Property Graph Pipeline

Understanding where these mechanisms apply clarifies Semantica's architecture:

1. Vector embedding storage — sqlite_vec_store.py protects FAISS index updates and metadata writes
2. Provenance tracking — provenance/storage.py isolates per-step metadata accumulation
3. Pipeline orchestration — resource_scheduler.py prevents resource reservation conflicts
4. Graph mutations — All store backends serialize writes to nodes, edges, and properties

Comparing Concurrency Strategies by Use Case

Scenario	Semantica's Approach	Key File
High-frequency writes to shared SQLite	threading.Lock + check_same_thread=False	sqlite_vec_store.py
Per-thread temporary buffers	threading.local()	provenance/storage.py
Recursive scheduling logic	threading.RLock()	pipeline/resource_scheduler.py
External database operations	Driver-native session isolation	graph_store/*_store.py

Summary

Semantica guarantees thread-safe in‑memory property graphs through:

• Explicit locks (threading.Lock) around all SQLite write operations in sqlite_vec_store.py
• Thread-local storage (threading.local) for buffer isolation in provenance tracking
• Re-entrant locks (threading.RLock) for safe recursive resource scheduling
• Driver-level session isolation ensuring no cross-thread session sharing in external graph stores
• SQLite connection flags (check_same_thread=False) enabling locked, shareable connections

This defense-in-depth approach lets Semantica safely operate concurrent pipelines on shared in‑memory graph structures without data corruption or performance bottlenecks.

Frequently Asked Questions

Does Semantica use a single global lock or fine-grained locking for its in‑memory graphs?

Semantica uses component-specific locks rather than a single global lock. The SQLite vector store maintains its own self._lock, the resource scheduler uses a separate RLock, and provenance storage avoids locks entirely via thread-local storage. This fine-grained approach minimizes contention between unrelated subsystems.

Why does Semantica set check_same_thread=False for SQLite connections?

The check_same_thread=False parameter disables SQLite's built-in thread-affinity check, which normally prohibits cross-thread connection usage. Semantica explicitly sets this flag only because it pairs it with explicit threading.Lock serialization—the lock guarantees actual safety, while the flag enables the connection-sharing pattern the architecture requires.

Can multiple threads safely read from the in‑memory graph without locking?

The current implementation in sqlite_vec_store.py uses a single threading.Lock for all database operations. While SQLite itself supports concurrent reads under certain configurations, Semantica's design prioritizes simplicity and correctness by serializing all access. Readers should not assume lock-free read access in production deployments.

How does thread safety differ between SQLite-backed storage and external stores like Neo4j?

SQLite-backed storage requires application-level locking because Python's SQLite driver lacks internal thread safety. External stores like Neo4j delegate threading concerns to their official drivers, which implement connection pooling and session isolation natively. Semantica respects this by creating fresh sessions per operation and never sharing driver objects across threads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}