Hybrid LSP Architecture for Python Type Resolution in codebase-memory-mcp

The codebase-memory-mcp repository implements a hybrid Language Server Protocol (LSP) architecture that combines fast, per-file Python analysis with a global CBMTypeRegistry to resolve types across files and languages.

The codebase-memory-mcp project handles Python type resolution through a sophisticated hybrid LSP architecture that bridges single-file analysis with repository-wide symbol resolution. This dual-track approach leverages both a pure-Python LSP evaluator for deterministic intra-file results and a cross-language registry for multi-file dependencies, as implemented in the core C source files.

How the Hybrid LSP Architecture Works

The hybrid LSP architecture in codebase-memory-mcp splits type resolution into two complementary mechanisms. First, a pure-Python LSP pass operates strictly within a single file using generated type information from python_stdlib_data.c. Second, a cross-file type registry (CBMTypeRegistry) stores all resolved types, functions, and classes, enabling later pipeline passes to merge Python results with other language LSPs.

Pure-Python LSP Pass

The per-file evaluator works through distinct phases (mirroring the "Python LSP plan" documented in internal/cbm/lsp/py_lsp.c). It processes imports, statements, and expressions to build a high-confidence baseline for call resolution without leaving the current file context. This phase uses only the generated std-lib data and local scope analysis.

Cross-File and Cross-Language Resolution

When the pure-Python pass cannot resolve a call, the system falls back to the global CBMTypeRegistry. This registry is populated by internal/cbm/lsp/generated/python_stdlib_data.c and extended during the cross-language merge phase in src/pipeline/pass_lsp_cross.c, allowing resolution of calls that span Python and JavaScript boundaries or require external module information.

Phase-by-Phase Type Resolution Pipeline

The Python LSP implementation in internal/cbm/lsp/py_lsp.c executes a structured pipeline that walks the AST in phases, binding symbols and evaluating types incrementally.

Phase 3 – Import Binding

The py_lsp_bind_imports function walks the import graph and populates the current scope with module symbols. Dotted prefixes (a, a.b, a.b.c) are bound via py_bind_dotted_prefixes, converting imports into MODULE or NAMED types that enable later attribute resolution.

Phase 4 – Statement Processing

During the statement walk, py_process_statement updates the per-file symbol table by processing assignments, for loops, with statements, and class or function definitions. This builds the local scope that the LSP evaluator queries during expression analysis.

Phase 5 – Expression Type Evaluation

The py_eval_expr_type function computes types for individual expressions with memoization per AST node. Uncached evaluation is handled by py_eval_expr_type_uncached. Results are cached and invalidated appropriately through py_scope_bind and py_scope_restore, ensuring consistent type inference while processing complex nested expressions.

Phase 6 – Attribute Lookup and Call Resolution

For attribute access, py_lookup_attribute follows Python's Method Resolution Order (MRO), falling back to the generated std-lib registry when necessary. The py_resolve_calls_in function orchestrates the recursive AST walk, emitting resolved calls with confidence scores via py_emit_resolved_call and py_emit_resolved_call_reason. This walk is depth-guarded by PY_LSP_MAX_EVAL_DEPTH to prevent stack overflow on complex codebases.

Key Source Files and Their Roles

The implementation spans several critical files:

• internal/cbm/lsp/py_lsp.c: Core Python LSP implementation containing phase handlers, scope management, and call emission logic.
• internal/cbm/lsp/generated/python_stdlib_data.c: Auto-generated type registry providing built-in and standard-library symbols for the resolver.
• src/pipeline/pass_lsp_cross.c: Cross-language merging pass that combines Python LSP results with TypeScript, Rust, and other language LSP outputs.
• tests/test_py_lsp.c: Unit tests verifying import resolution, built-in identification, and attribute chain handling.
• scripts/gen-py-stdlib.py: Generation script for creating the std-lib registry data.

Practical Usage Example

The following C code demonstrates initializing a Python LSP context and resolving calls:

/* Initialise a Python LSP context for a single source file */
PyLSPContext ctx;
py_lsp_init(&ctx,
            arena,                     // CBMArena for temporary allocations
            source, source_len,        // the Python source text
            registry,                  // CBMTypeRegistry with std‑lib data
            "my_pkg.my_module",        // module qualified name
            &resolved_calls);          // CBMResolvedCallArray to collect results

/* Register imports that were extracted earlier by the extractor */
py_lsp_add_import(&ctx, "os", "os");
py_lsp_add_import(&ctx, "Path", "pathlib.Path");

/* Bind those imports into the scope (Phase 3) */
py_lsp_bind_imports(&ctx);

/* Walk the AST (Phase 4‑6) and emit resolved calls */
py_resolve_calls_in(&ctx, root_node);

/* After the walk you can inspect `resolved_calls` for the LSP‑resolved entries */
for (size_t i = 0; i < resolved_calls.count; ++i) {
    const CBMResolvedCall *rc = &resolved_calls.items[i];
    printf("Call %zu: %s → %s (confidence %.2f, strategy %s)\n",
           i, rc->caller_qn, rc->callee_qn, rc->confidence, rc->strategy);
}

Testing the resolver on a simple import scenario, as shown in tests/test_py_lsp.c:

/* In test_py_lsp.c – a simplified excerpt */
static void test_simple_import(void) {
    const char *src = "import os\nos.path.join('a', 'b')\n";
    CBMResolvedCallArray out = {0};
    PyLSPContext ctx;
    py_lsp_init(&ctx, arena, src, strlen(src), registry,
                "test.mod", &out);
    py_lsp_add_import(&ctx, "os", "os");
    py_lsp_bind_imports(&ctx);
    py_resolve_calls_in(&ctx, parse(src));
    assert(out.count == 1);
    assert(strcmp(out.items[0].callee_qn,
                  "os.path.join") == 0);
}

Summary

• The hybrid LSP architecture combines per-file Python analysis with a global CBMTypeRegistry for cross-file resolution.
• Resolution occurs in six phases: Import Binding, Statement Processing, Expression Evaluation, Attribute Lookup, and Call Resolution.
• Key functions include py_lsp_bind_imports, py_eval_expr_type, and py_resolve_calls_in, all implemented in internal/cbm/lsp/py_lsp.c.
• The system uses memoization and scope caching to optimize performance while maintaining depth guards against stack overflow.
• Cross-language merging in src/pipeline/pass_lsp_cross.c enables the system to resolve calls between Python and other languages.

Frequently Asked Questions

What makes the LSP architecture "hybrid" in codebase-memory-mcp?

The architecture is hybrid because it operates on two levels simultaneously: a fast, deterministic pure-Python LSP pass that resolves symbols within a single file, and a cross-file registry that merges results across the entire repository and multiple languages. This allows the system to provide high-confidence results for local code while maintaining the ability to resolve inter-file and cross-language dependencies.

How does the system handle Python standard library types?

Standard library types are resolved through the CBMTypeRegistry, which is pre-populated by the auto-generated file internal/cbm/lsp/generated/python_stdlib_data.c. When the attribute lookup in py_lookup_attribute encounters a built-in or std-lib symbol, it queries this registry to resolve the type without requiring external analysis.

What prevents infinite recursion during type resolution?

The system implements a depth-guarded walk using the PY_LSP_MAX_EVAL_DEPTH constant. The py_resolve_calls_in function tracks evaluation depth during recursive AST traversal, halting resolution if the depth exceeds the configured limit to prevent stack overflow when processing complex circular imports or deeply nested expressions.

How are the results from the Python LSP pass integrated with other languages?

After the Python LSP completes its per-file analysis, the src/pipeline/pass_lsp_cross.c module merges the resolved calls with results from other language LSPs (such as TypeScript or Rust). This merge process combines confidence scores and strategy tags, enabling the system to resolve calls where a Python function invokes a JavaScript API or vice versa.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}