BM25 Full-Text Search Implementation with CamelCase/SnakeCase Tokenizer in codebase-memory-mcp

The codebase-memory-mcp repository implements BM25-ranked full-text search using SQLite's FTS5 extension with a custom cbm_camel_split tokenizer that splits identifiers on case changes and underscores, enabling partial matches for individual words within camelCase and snake_case tokens.

The codebase-memory-mcp project provides intelligent code search capabilities by combining SQLite's FTS5 full-text search engine with a custom tokenizer designed specifically for source code identifiers. This implementation leverages the BM25 ranking algorithm to deliver relevant results while understanding the naming conventions common in programming languages. By registering the cbm_camel_split tokenizer, the system breaks compound identifiers like myFunctionName or my_function_name into searchable lexical components.

BM25 Search Architecture Overview

The full-text search implementation relies on SQLite's FTS5 virtual table mechanism with the BM25 ranking function enabled. According to the source code in src/store/store.c, the system creates a virtual table at line 326 that stores indexed source code content for BM25 scoring.

The architecture consists of three primary components:

• FTS5 Virtual Table: Stores tokenized source code content in src/store/store.c
• Custom Tokenizer: The cbm_camel_split module registered in src/mcp/mcp.c at line 367
• Query Executor: BM25 query path implementation spanning lines 2233-2691 in src/mcp/mcp.c

The cbm_camel_split Tokenizer Implementation

The custom tokenizer is the core innovation that enables effective code search. Registered in src/mcp/mcp.c at line 367, the cbm_camel_split tokenizer processes text streams by detecting transitions between lowercase and uppercase letters, as well as underscore characters.

When indexing content, the tokenizer emits sub-tokens for compound identifiers:

• myFunctionName produces my, function, name
• my_function_name produces my, function, name

This behavior ensures that searches for function or name match complete identifiers containing those terms, significantly improving recall for code-specific queries.

Indexing and Query Execution

Creating the FTS5 Virtual Table

The virtual table creation occurs in src/store/store.c at line 326. The system initializes the FTS5 table with the custom tokenizer specified:

/* src/store/store.c */
sqlite3_exec(db,
    "CREATE VIRTUAL TABLE IF NOT EXISTS src_fts USING fts5("
    "content, tokenize = 'cbm_camel_split');",
    NULL, NULL, NULL);

Registering the Custom Tokenizer

The tokenizer registration happens in src/mcp/mcp.c at line 367. The implementation uses a registration function to make the custom tokenizer available to SQLite:

/* src/mcp/mcp.c */
static int register_camel_split_tokenizer(sqlite3 *db) {
    return sqlite3_create_tokenizer(db, "cbm_camel_split", &camel_split_module, NULL);
}

/* Called during initialization */
register_camel_split_tokenizer(db);

Executing BM25 Queries

The BM25 query path, implemented in src/mcp/mcp.c between lines 2233 and 2691, executes full-text searches using the BM25 ranking function. The query construction follows this pattern:

SELECT rowid, bm25(my_fts) AS rank, snippet(my_fts) 
FROM my_fts 
WHERE my_fts MATCH ? 
ORDER BY rank;

The placeholder receives the user's query string, which undergoes the same tokenization process as the indexed content, ensuring consistent term matching. SQLite's built-in BM25 algorithm calculates relevance scores based on term frequency, inverse document frequency, and document length.

From the CLI, users can trigger this search path:

./codebase-memory-mcp query '{"type":"bm25","query":"functionName"}'

Build Configuration

Enabling FTS5 support requires specific compilation flags. In Makefile.cbm at line 293, the build system defines SQLITE_ENABLE_FTS5:

# Makefile.cbm

CFLAGS += -DSQLITE_ENABLE_FTS5

This flag ensures the SQLite library includes the FTS5 extension required for the virtual table and BM25 functionality.

Summary

• The BM25 full-text search implementation in codebase-memory-mcp uses SQLite FTS5 with a custom cbm_camel_split tokenizer to handle code-specific naming conventions.
• The tokenizer is registered in src/mcp/mcp.c at line 367 and splits both camelCase and snake_case identifiers into searchable sub-tokens.
• FTS5 virtual tables are created in src/store/store.c at line 326 to store indexed content with BM25 ranking capabilities.
• The BM25 query execution path spans lines 2233-2691 in src/mcp/mcp.c, utilizing SQLite's built-in BM25 function for relevance scoring.
• Compilation requires the SQLITE_ENABLE_FTS5 flag defined in Makefile.cbm at line 293.

Frequently Asked Questions

What is BM25 and why is it used for code search?

BM25 is a probabilistic ranking function used by search engines to estimate the relevance of documents based on query terms. In codebase-memory-mcp, BM25 provides superior ranking compared to simple term matching because it considers term frequency and document length, ensuring that files containing more relevant mentions of search terms appear higher in results.

How does the cbm_camel_split tokenizer handle mixed case identifiers?

The tokenizer processes text streams by detecting transitions between lowercase and uppercase letters and splitting on underscore characters. For example, XMLParser would emit xml and parser, while parseXMLDocument would emit parse, xml, and document. This ensures that searching for any component of a compound identifier returns the complete match.

Can I query the FTS5 virtual table directly with standard SQL?

Yes, the FTS5 virtual table supports standard SQL queries with the MATCH operator for full-text search. You can query the src_fts table directly using SELECT * FROM src_fts WHERE src_fts MATCH 'functionName', though the application typically routes these through the BM25 ranking path for ordered results.

Where is the tokenizer initialized during application startup?

The cbm_camel_split tokenizer is registered during the initialization phase in src/mcp/mcp.c at line 367. The register_camel_split_tokenizer function calls the SQLite tokenizer registration API, making the custom tokenizer available before any indexing or search operations occur.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}