Daemon Coordination System Architecture in codebase-memory-mcp: Three-Layer Design Explained

The daemon coordination system in src/daemon implements a three-layer architecture—Service, Runtime, and IPC—that enforces build-identity compatibility through a HELLO handshake, manages client leases via heartbeat timeouts, and supports cross-generation graceful shutdown without requiring full session authentication.

The DeusData/codebase-memory-mcp repository contains a sophisticated daemon coordination system in src/daemon designed to manage IPC transport, version compatibility, and client lifecycle for long-running background processes. This architecture ensures that only compatible build versions can establish communication while enabling safe upgrades through deterministic conflict detection and graceful handoff mechanisms.

Three-Layer Architecture Overview

The daemon coordination system organizes functionality into three distinct logical layers, each with specific responsibilities and well-defined interfaces.

Service Layer: Policy and Identity

The Service layer, defined in src/daemon/service.h and implemented in service.c, manages the immutable aspects of daemon identity and compatibility checking. According to the DeusData/codebase-memory-mcp source code, this layer holds the stable rendezvous key and cbm_daemon_build_identity_t structures that uniquely identify each daemon instance through semantic versions, build fingerprints, and ABI levels.

Key responsibilities include:

• Storing the build identity (semantic version, 64-character hex SHA-256 build fingerprint, cache fingerprint, and ABI versions)
• Implementing cbm_daemon_hello_compare() to determine compatibility between client and daemon identities
• Generating human-readable conflict messages via cbm_daemon_conflict_format()
• Logging version conflicts to persistent storage using cbm_daemon_conflict_log_append()

Runtime Layer: Connection and Request Handling

The Runtime layer, located in src/daemon/runtime.h and runtime.c, constitutes the core orchestration engine. This layer implements the cbm_daemon_runtime_service_config_t configuration structure and manages the complete client lifecycle from admission to disconnection.

The runtime handles:

• Admission flow: Processing HELLO requests and validating build identities against the service layer
• Client state: Maintaining cbm_daemon_runtime_client_t objects that store client_id, authenticated kernel PID, and active build fingerprints
• Operation dispatch: Routing one-byte operation codes (cbm_daemon_runtime_operation_t) to appropriate handlers
• Lease management: Enforcing lease_timeout_ms deadlines that automatically disconnect unresponsive clients

IPC Layer: Transport Plumbing

The IPC layer (src/daemon/ipc.h and ipc.c) provides platform-specific abstraction for endpoint creation and low-level communication. This layer creates cbm_daemon_ipc_endpoint_t instances representing ACL-restricted UNIX sockets or Windows named pipes that isolate communication to the current OS account.

Functions like cbm_daemon_ipc_endpoint_create() handle lifetime reservation and cleanup, while the layer exposes send/receive primitives used by the runtime for all client-daemon communication.

Admission Flow and Build Identity Verification

When a client initiates communication, the daemon coordination system enforces strict version compatibility through a deterministic admission sequence.

The process begins with cbm_daemon_runtime_client_connect(), which transmits a fixed-size HELLO request encoded by cbm_daemon_runtime_hello_request_encode(). The daemon's service layer then executes cbm_daemon_hello_compare() to evaluate compatibility, returning one of the cbm_daemon_hello_status_t values:

• Compatible: Build fingerprints and ABI versions match
• Version conflict: Semantic version mismatch
• Build conflict: Different build fingerprints
• ABI conflicts: Protocol, store, or feature ABI mismatch
• Cache conflict: Cache fingerprint inconsistency

If conflicts exist, the system generates diagnostic output through cbm_daemon_conflict_format() and appends records via cbm_daemon_conflict_log_append() before rejecting the connection.

Client Lifecycle and Operation Dispatch

Once admitted, clients enter a managed lifecycle governed by the runtime layer's operation dispatch system.

Client Object Creation

Successful admission creates a cbm_daemon_runtime_client_t instance containing:

• Unique client_id assigned by the runtime
• Authenticated PID extracted from kernel credentials
• Reference to the service layer for status queries
• Lease timeout tracking mechanism

Operation Codes and Routing

Every client request begins with a one-byte operation code (cbm_daemon_runtime_operation_t) that determines handling:

• HELLO: Initial admission (only valid during connection setup)
• HEARTBEAT: Resets the lease_timeout_ms timer to maintain connectivity
• JOB_SUBSCRIBE / JOB_UNSUBSCRIBE: Manages asynchronous job notification subscriptions
• APPLICATION_REQUEST / APPLICATION_REQUEST_TAGGED: Invokes user-provided callbacks (session_open, request, request_cancel) defined in the service configuration struct (lines 70-76 of runtime.h)
• STATUS / STOP: One-shot control operations that function across build skews without requiring full session authentication

The runtime validates payload lengths—fixed-size for most operations, length-prefixed for application requests—before routing to handlers.

Lease-Based Resource Management

The coordination system implements coordinator leases to prevent resource exhaustion. Each client must invoke cbm_daemon_runtime_client_heartbeat() within the lease_timeout_ms window (default typically 30,000ms). Failure to maintain heartbeats triggers automatic connection closure, freeing slots for new clients according to the max_clients limit specified in cbm_daemon_runtime_service_config_t.

Graceful Shutdown and Activation Control

The architecture supports multi-generational daemon management through special activation operations that bypass standard admission requirements.

Operations like ACTIVATION_SHUTDOWN, STOP, and STATUS do not require a full HELLO handshake, allowing newer daemon generations to communicate with older instances. This enables:

• Graceful draining: New daemons request ACTIVATION_SHUTDOWN to signal older instances to stop accepting new connections while completing active work
• Coordinated upgrades: Build-differentiated instances can negotiate state handoffs
• Emergency termination: Direct stop requests via cbm_daemon_runtime_request_activation_shutdown()

The function cbm_daemon_runtime_request_activation_shutdown() accepts a timeout parameter (shutdown_timeout_ms) and returns a cbm_daemon_runtime_activation_result_t indicating whether the target daemon accepted the activation request.

Key Source Files and Responsibilities

File	Primary Role	Key Components
src/daemon/service.h / service.c	Build identity and conflict handling	cbm_daemon_build_identity_t, cbm_daemon_hello_compare(), conflict logging
src/daemon/runtime.h / runtime.c	Client lifecycle and operation dispatch	cbm_daemon_runtime_service_config_t, cbm_daemon_runtime_client_t, operation codes
src/daemon/ipc.h / ipc.c	Transport abstraction	cbm_daemon_ipc_endpoint_t, endpoint creation, send/receive primitives
src/daemon/project_lock.h / project_lock.c	Singleton enforcement	Global lock ensuring one daemon per OS account per endpoint
src/daemon/version_cohort.h / version_cohort.c	Version grouping	Helpers for aggregating daemon instances by build fingerprints

Implementation Examples

Starting a Daemon Service

/* Build identity of this daemon (filled once at startup) */
cbm_daemon_build_identity_t my_identity = {
    .semantic_version   = "1.2.3",
    .build_fingerprint  = "a3f5…",            // 64‑char hex SHA‑256
    .cache_fingerprint  = NULL,               // optional
    .protocol_abi       = 1,
    .store_abi          = 1,
    .feature_abi        = 1,
};

/* Create IPC endpoint (UNIX socket or Windows named pipe) */
cbm_daemon_ipc_endpoint_t *ep = cbm_daemon_ipc_endpoint_create("cbm-daemon.sock");

/* Configure the runtime */
cbm_daemon_runtime_service_config_t cfg = {
    .endpoint                = ep,
    .identity                = my_identity,
    .conflict_log_path       = "/var/log/cbm/conflict.log",
    .conflict_log_cap_bytes  = 64 * 1024,
    .max_clients             = 32,
    .lease_timeout_ms       = 30000,
    .request_timeout_ms      = 10000,
    .shutdown_timeout_ms    = 60000,
    .application             = {/* all callbacks NULL → daemon runs without app layer */},
    .permanent               = false,
};

/* Start the service (blocks until the endpoint is bound) */
cbm_daemon_runtime_service_t *svc = cbm_daemon_runtime_service_start(&cfg);
if (!svc) { /* handle error */ }

Connecting a Client to the Daemon

cbm_daemon_build_identity_t client_id = {
    .semantic_version   = "1.2.3",
    .build_fingerprint  = "a3f5…",
    .cache_fingerprint  = NULL,
    .protocol_abi       = 1,
    .store_abi          = 1,
    .feature_abi       = 1,
};

cbm_daemon_runtime_connect_result_t result;
cbm_daemon_runtime_client_t *client = cbm_daemon_runtime_client_connect(
    ep, &client_id, /*timeout_ms=*/5000, &result);

if (!client) {
    printf("Connection failed: %s\n", result.message);
    exit(1);
}

/* Send a heartbeat to keep the lease alive */
bool ok = cbm_daemon_runtime_client_heartbeat(client, 5000);

Issuing an Application Request

/* Assume callbacks have been supplied in the service config */
uint8_t *resp = NULL;
uint32_t resp_len = 0;
cbm_daemon_runtime_application_status_t status =
    cbm_daemon_runtime_client_application_request(
        client,
        /*request*/ my_payload, (uint32_t)my_payload_len,
        &resp, &resp_len,
        /*timeout_ms=*/5000);

if (status == CBM_DAEMON_RUNTIME_APPLICATION_OK) {
    // Process response (resp is malloc‑owned)
    free(resp);
}

Graceful Shutdown of a Running Daemon

cbm_daemon_runtime_activation_result_t act_res;
bool ok = cbm_daemon_runtime_request_activation_shutdown(
    ep, &my_identity,
    CBM_DAEMON_RUNTIME_ACTIVATION_SHUTDOWN,   // or INSTALL/UPDATE/UNINSTALL
    30000,
    &act_res);

if (ok && act_res.accepted) {
    printf("Daemon drained successfully.\n");
}

Summary

• The daemon coordination system employs a three-layer architecture (Service, Runtime, IPC) that separates identity policy from connection management and transport mechanics.
• Build identity verification occurs through the HELLO handshake and cbm_daemon_hello_compare(), preventing incompatible clients from establishing sessions.
• Lease-based heartbeats (cbm_daemon_runtime_client_heartbeat()) enforce resource limits and automatically reclaim stale connections.
• Cross-generation communication via ACTIVATION_SHUTDOWN and STOP operations enables zero-downtime upgrades without requiring full client authentication.
• The IPC layer enforces OS-account isolation through ACL-restricted endpoints, ensuring only processes owned by the same account can communicate with the daemon.

Frequently Asked Questions

How does the daemon coordination system handle version conflicts between clients and the server?

When a client connects, the runtime layer invokes cbm_daemon_hello_compare() (defined in service.c) to compare the client's cbm_daemon_build_identity_t against the daemon's active identity. The function returns specific conflict types—version conflict, build conflict, ABI conflicts, or cache conflict—and automatically generates human-readable diagnostic messages via cbm_daemon_conflict_format(), logging them to the configured conflict log path while rejecting the connection.

What is the purpose of the HELLO handshake in the daemon admission process?

The HELLO handshake serves as a version compatibility gate where cbm_daemon_runtime_hello_request_encode() transmits the client's build fingerprint and ABI levels to the daemon before establishing a full session. This fixed-size request allows the service layer to validate identity through cbm_daemon_hello_compare() without allocating per-client resources for incompatible connections, ensuring that only matching build versions can proceed to create cbm_daemon_runtime_client_t objects.

How does the runtime layer prevent resource exhaustion from idle client connections?

The runtime implements coordinator leases using the lease_timeout_ms parameter specified in cbm_daemon_runtime_service_config_t. Each successful admission creates a client with an active lease timer that must be reset via HEARTBEAT operations (invoked through cbm_daemon_runtime_client_heartbeat()). If a client fails to maintain heartbeats within the timeout window, the runtime automatically closes the connection and frees the slot, respecting the max_clients limit to prevent file descriptor exhaustion.

What mechanism allows new daemon versions to gracefully terminate older instances?

The system supports activation operations (ACTIVATION_SHUTDOWN, STOP) that bypass the standard HELLO admission requirements, allowing newer daemon generations to communicate with older instances despite build fingerprint differences. By calling cbm_daemon_runtime_request_activation_shutdown() with the target endpoint and a shutdown timeout, administrators can request that an older daemon drain active connections and terminate, enabling seamless upgrades without interrupting in-flight operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}