How the Project Lock Mechanism Works in DeusData/codebase-memory-mcp

The project lock mechanism provides a coordinated, process-local lease system that protects mutations of named projects through a three-layer stack comprising OS file locks, a FIFO lock registry, and a project-lock manager.

The codebase-memory-mcp repository from DeusData implements this locking subsystem to serialize access between daemon processes and CLI clients. By combining a shared project-set lock with an exclusive per-project lock, the mechanism prevents race conditions while allowing concurrent operations on different projects. The entire architecture operates within a single process boundary, using private directories and side-car lock files to guarantee safety without requiring kernel-level semaphores.

Architecture Overview

The project lock mechanism is built on a hierarchical stack of three primitives that transform low-level OS file locks into high-level project leases:

Layer	Source File	Responsibility
Private File Lock	src/foundation/private_file_lock.h	Wraps native SH (shared) and EX (exclusive) OS locks, ensuring lock files reside inside a validated private directory and handling error normalization.
Lock Registry	src/foundation/lock_registry.c	Maintains a FIFO, writer-preference queue that maps logical resource keys to side-car lock files (*.turn and *.rw), implementing the turn-based acquisition protocol.
Project-Lock Manager	src/daemon/project_lock.c	Provides the façade cbm_project_lock_manager_t, normalizes project names into deterministic keys, and composes two registry leases into a single project-wide lease.

The Three-Layer Lock Stack

Private File Lock Primitives

At the foundation, the system uses src/foundation/private_file_lock.h to interact with the operating system’s file locking facilities. This layer abstracts away platform-specific differences and enforces that all lock files remain within a pre-validated private directory created via cbm_daemon_ipc_private_lock_directory_new. The interface exposes cbm_private_file_lock_try_acquire() for non-blocking attempts and cbm_private_file_lock_release() for cleanup, returning standardized status codes that higher layers consume.

The Lock Registry

The lock registry in src/foundation/lock_registry.c transforms the primitive file locks into a logical locking service. For each resource key (e.g., a normalized project name), the registry creates two side-car files: a *.turn file that governs the wait queue and a *.rw file that holds the actual shared or exclusive lock.

When cbm_lock_registry_acquire() is invoked, the registry executes a turn-based protocol:

1. Queue insertion: The waiter is pushed onto a FIFO queue (lock_registry_waiter_push).
2. Turn lock acquisition: The thread attempts to acquire the native lock on the *.turn file (lines 53‑58).
3. RW lock acquisition: The thread attempts to acquire the native lock on the *.rw file (lines 79‑84).
4. Shared lock optimization: If the request is for a shared lock, the turn lock is released immediately (lines 95‑99) to allow parallel readers.
5. Lease activation: Once both locks are held, the registry updates internal counters (active_lease_count, waiter_count) and returns the lease structure (lines 138‑148).

The registry supports cancellation tokens (cbm_lock_cancel_token_t) and absolute deadlines (uint64_t deadline_ms). During the parking/wait loop, lock_registry_should_stop (lines 87‑92) checks these constraints and aborts with CBM_PRIVATE_FILE_LOCK_BUSY if the token is set or the deadline expires.

Project-Lock Manager

The src/daemon/project_lock.c file implements the high-level interface used by the daemon and CLI. The cbm_project_lock_manager_new() function (lines 45‑48) initializes the manager by opening the private lock directory and instantiating a cbm_lock_registry_t (line 57) that persists for the process lifetime.

The manager normalizes project names through project_lock_key() (lines 23‑41), which:

• Rejects NULL, empty strings, and wildcards ("*").
• Prepends the literal prefix "cbm-project-v1:".
• Converts the project name to lowercase for case-insensitive matching.

Acquiring a Project Lock

Normalizing Project Names

Before any lock operation, the project name is transformed into a deterministic resource key. As implemented in src/daemon/project_lock.c, the normalization ensures that "MyProject" and "myproject" map to the same underlying lock key "cbm-project-v1:myproject". This prevents deadlocks from case mismatches and reserves the "cbm-project-v1:" namespace for the mechanism.

Two-Stage Acquisition

cbm_project_lock_acquire() (lines 54‑60) delegates to project_lock_acquire_internal() (lines 9‑52), which acquires locks in two strict stages:

1. Project-set lock: Acquires a shared lock (SH) on the global key PROJECT_SET_KEY ("cbm-project-set-v1", line 11). For wildcard requests, this becomes an exclusive lock (EX).
2. Individual project lock: Acquires an exclusive lock (EX) on the normalized project-specific key.

Both acquisitions route through cbm_lock_registry_acquire() (or cbm_lock_registry_try_acquire() for the non-blocking variant). If either stage fails, project_lock_failed_acquire() (lines 93‑107) releases any partially held locks and returns the error status.

The Turn-Based Protocol

The registry’s turn-based protocol ensures writer preference while preventing starvation. By requiring waiters to acquire the turn lock before the rw lock, the system serializes access to the wait queue. Once a writer holds the turn lock, no new readers can enter the queue, ensuring that writers eventually obtain the rw lock. For shared locks, the turn lock is released immediately after acquiring the rw lock, allowing concurrent shared access.

Releasing Leases and Cancellation

Releasing a project lock requires tearing down the two-stage acquisition in reverse order. cbm_project_lock_lease_release() (lines 67‑90) performs:

1. Release of the individual project lease via cbm_lock_lease_release(&lease->project).
2. Release of the project-set lease via cbm_lock_lease_release(&lease->project_set).
3. Deallocation of the cbm_project_lock_lease_t structure (lines 88‑89).

Cancellation is cooperative. Callers initialize a cbm_lock_cancel_token_t (a boolean flag) and pass it to cbm_project_lock_acquire(). To cancel a pending acquisition from another thread, set the token to true and invoke cbm_project_lock_request_cancel(). The registry polls this token in the wait loop and aborts with CBM_PRIVATE_FILE_LOCK_BUSY.

Code Examples

Blocking Acquisition

The standard pattern for acquiring exclusive access to a project uses a deadline and cancellation token:

cbm_project_lock_manager_t *mgr = cbm_project_lock_manager_new(endpoint);
if (!mgr) { /* handle initialization error */ }

cbm_project_lock_lease_t *lease = NULL;
cbm_lock_cancel_token_t token = false;
uint64_t deadline = cbm_now_ms() + 5000;  // 5-second timeout

cbm_private_file_lock_status_t st = cbm_project_lock_acquire(
    mgr,
    "my-awesome-project",
    deadline,
    &token,
    &lease);

if (st == CBM_PRIVATE_FILE_LOCK_OK) {
    /* Critical section: exclusive access to the project */
    perform_project_mutation();
    
    cbm_project_lock_lease_release(&lease);
}

This call internally creates the normalized key "cbm-project-v1:my-awesome-project" and obtains both the shared project-set lock and the exclusive project lock.

Non-Blocking Try-Acquire

For scenarios where waiting is unacceptable, use the non-blocking variant:

cbm_project_lock_lease_t *lease = NULL;
cbm_private_file_lock_status_t st = cbm_project_lock_try_acquire(
    mgr, 
    "my-awesome-project", 
    &lease);

if (st == CBM_PRIVATE_FILE_LOCK_OK) {
    /* Obtained lock immediately */
    perform_operation();
    cbm_project_lock_lease_release(&lease);
} else if (st == CBM_PRIVATE_FILE_LOCK_BUSY) {
    /* Lock held elsewhere, proceed with fallback logic */
}

Cancellation with Deadlines

To implement timeouts or user-interruptible operations:

cbm_lock_cancel_token_t token = false;
cbm_project_lock_lease_t *lease = NULL;
uint64_t deadline = cbm_now_ms() + 10000;  // 10-second deadline

/* This call blocks until acquisition or cancellation */
cbm_private_file_lock_status_t st = cbm_project_lock_acquire(
    mgr, "my-awesome-project", deadline, &token, &lease);

/* From another thread or signal handler: */
atomic_store_explicit(&token, true, memory_order_release);
cbm_project_lock_request_cancel(mgr, &token);

Testing and Fault Injection

The codebase-memory-mcp test suite validates the project lock mechanism through targeted fault injection. The lock registry exposes test-only hooks such as cbm_lock_registry_fail_next_native_release_step_for_test() and cbm_lock_registry_set_stage_hook_for_test() that allow tests in tests/test_lock_registry.c to simulate native OS errors and verify cleanup paths.

The high-level logic is exercised in tests/test_project_lock.c, which validates correct acquisition ordering, deadline handling, and cancellation semantics. These tests ensure that partial acquisitions (where the project-set lock succeeds but the project lock fails) correctly release held resources via project_lock_failed_acquire().

Summary

• The project lock mechanism uses a three-layer stack: OS file locks, a FIFO lock registry with writer preference, and a project-lock manager façade.
• Two-stage acquisition first locks the global project-set (shared) then the individual project (exclusive), preventing cross-project deadlocks while ensuring isolation.
• Turn-based protocol in src/foundation/lock_registry.c uses side-car *.turn and *.rw files to queue waiters and grant locks atomically.
• Cancellation and deadlines are supported throughout via cbm_lock_cancel_token_t and absolute millisecond timestamps checked in the registry’s wait loops.
• Case-insensitive normalization converts project names to lowercase with a "cbm-project-v1:" prefix to ensure consistent resource keys.
• Fault injection hooks in the registry enable comprehensive testing of error paths and resource cleanup in tests/test_project_lock.c.

Frequently Asked Questions

What is the difference between the project-set lock and the individual project lock?

The project-set lock (key "cbm-project-set-v1") governs access to the collection of all projects; it is acquired in shared mode (SH) for normal project operations and exclusive mode (EX) for wildcard operations. The individual project lock uses the normalized project-specific key (e.g., "cbm-project-v1:myproject") and is always acquired in exclusive mode (EX) to ensure only one writer modifies a specific project at a time.

How does cancellation work in the project lock mechanism?

Cancellation is cooperative. The caller provides a cbm_lock_cancel_token_t (a boolean flag) to cbm_project_lock_acquire(). The registry periodically checks this token in lock_registry_should_stop() (lines 87‑92 of src/foundation/lock_registry.c). Setting the token to true via atomic_store_explicit and calling cbm_project_lock_request_cancel() causes any waiting acquisition to abort and return CBM_PRIVATE_FILE_LOCK_BUSY.

Can the project lock mechanism work across multiple processes?

No. According to the source architecture in src/daemon/project_lock.c, the lock mechanism is process-local. The cbm_project_lock_manager_t and its underlying cbm_lock_registry_t exist within a single process, coordinating threads rather than separate processes. Cross-process coordination would require a different transport mechanism (e.g., the daemon IPC endpoint passed to cbm_project_lock_manager_new()).

Where are the lock files physically stored?

Lock files reside in a private lock directory returned by cbm_daemon_ipc_private_lock_directory_new() during manager initialization (lines 45‑48 of src/daemon/project_lock.c). This directory contains side-car files with extensions *.turn and *.rw for each resource key, ensuring lock files are isolated from user data and have appropriate filesystem permissions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}