# BM25 Full-Text Search Implementation with CamelCase/SnakeCase Tokenizer in codebase-memory-mcp

> Implement BM25 full-text search with a custom camelCase/snake_case tokenizer in codebase-memory-mcp. Achieve partial word matches in identifiers using SQLite FTS5.

- Repository: [Martin Vogel/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)
- Tags: deep-dive
- Published: 2026-07-16

---

**The codebase-memory-mcp repository implements BM25-ranked full-text search using SQLite's FTS5 extension with a custom `cbm_camel_split` tokenizer that splits identifiers on case changes and underscores, enabling partial matches for individual words within camelCase and snake_case tokens.**

The codebase-memory-mcp project provides intelligent code search capabilities by combining SQLite's FTS5 full-text search engine with a custom tokenizer designed specifically for source code identifiers. This implementation leverages the BM25 ranking algorithm to deliver relevant results while understanding the naming conventions common in programming languages. By registering the `cbm_camel_split` tokenizer, the system breaks compound identifiers like `myFunctionName` or `my_function_name` into searchable lexical components.

## BM25 Search Architecture Overview

The full-text search implementation relies on SQLite's FTS5 virtual table mechanism with the BM25 ranking function enabled. According to the source code in [`src/store/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/store/store.c), the system creates a virtual table at line 326 that stores indexed source code content for BM25 scoring.

The architecture consists of three primary components:

- **FTS5 Virtual Table**: Stores tokenized source code content in [`src/store/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/store/store.c)
- **Custom Tokenizer**: The `cbm_camel_split` module registered in [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c) at line 367
- **Query Executor**: BM25 query path implementation spanning lines 2233-2691 in [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c)

## The `cbm_camel_split` Tokenizer Implementation

The custom tokenizer is the core innovation that enables effective code search. Registered in [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c) at line 367, the `cbm_camel_split` tokenizer processes text streams by detecting transitions between lowercase and uppercase letters, as well as underscore characters.

When indexing content, the tokenizer emits sub-tokens for compound identifiers:

- `myFunctionName` produces `my`, `function`, `name`
- `my_function_name` produces `my`, `function`, `name`

This behavior ensures that searches for `function` or `name` match complete identifiers containing those terms, significantly improving recall for code-specific queries.

## Indexing and Query Execution

### Creating the FTS5 Virtual Table

The virtual table creation occurs in [`src/store/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/store/store.c) at line 326. The system initializes the FTS5 table with the custom tokenizer specified:

```c
/* src/store/store.c */
sqlite3_exec(db,
    "CREATE VIRTUAL TABLE IF NOT EXISTS src_fts USING fts5("
    "content, tokenize = 'cbm_camel_split');",
    NULL, NULL, NULL);

```

### Registering the Custom Tokenizer

The tokenizer registration happens in [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c) at line 367. The implementation uses a registration function to make the custom tokenizer available to SQLite:

```c
/* src/mcp/mcp.c */
static int register_camel_split_tokenizer(sqlite3 *db) {
    return sqlite3_create_tokenizer(db, "cbm_camel_split", &camel_split_module, NULL);
}

/* Called during initialization */
register_camel_split_tokenizer(db);

```

### Executing BM25 Queries

The BM25 query path, implemented in [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c) between lines 2233 and 2691, executes full-text searches using the BM25 ranking function. The query construction follows this pattern:

```sql
SELECT rowid, bm25(my_fts) AS rank, snippet(my_fts) 
FROM my_fts 
WHERE my_fts MATCH ? 
ORDER BY rank;

```

The placeholder receives the user's query string, which undergoes the same tokenization process as the indexed content, ensuring consistent term matching. SQLite's built-in BM25 algorithm calculates relevance scores based on term frequency, inverse document frequency, and document length.

From the CLI, users can trigger this search path:

```bash
./codebase-memory-mcp query '{"type":"bm25","query":"functionName"}'

```

## Build Configuration

Enabling FTS5 support requires specific compilation flags. In `Makefile.cbm` at line 293, the build system defines `SQLITE_ENABLE_FTS5`:

```makefile

# Makefile.cbm

CFLAGS += -DSQLITE_ENABLE_FTS5

```

This flag ensures the SQLite library includes the FTS5 extension required for the virtual table and BM25 functionality.

## Summary

- The **BM25 full-text search implementation** in codebase-memory-mcp uses SQLite FTS5 with a custom `cbm_camel_split` tokenizer to handle code-specific naming conventions.
- The tokenizer is registered in **[`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c)** at line 367 and splits both camelCase and snake_case identifiers into searchable sub-tokens.
- **FTS5 virtual tables** are created in [`src/store/store.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/store/store.c) at line 326 to store indexed content with BM25 ranking capabilities.
- The **BM25 query execution path** spans lines 2233-2691 in [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c), utilizing SQLite's built-in BM25 function for relevance scoring.
- Compilation requires the **`SQLITE_ENABLE_FTS5`** flag defined in `Makefile.cbm` at line 293.

## Frequently Asked Questions

### What is BM25 and why is it used for code search?

BM25 is a probabilistic ranking function used by search engines to estimate the relevance of documents based on query terms. In codebase-memory-mcp, BM25 provides superior ranking compared to simple term matching because it considers term frequency and document length, ensuring that files containing more relevant mentions of search terms appear higher in results.

### How does the cbm_camel_split tokenizer handle mixed case identifiers?

The tokenizer processes text streams by detecting transitions between lowercase and uppercase letters and splitting on underscore characters. For example, `XMLParser` would emit `xml` and `parser`, while `parseXMLDocument` would emit `parse`, `xml`, and `document`. This ensures that searching for any component of a compound identifier returns the complete match.

### Can I query the FTS5 virtual table directly with standard SQL?

Yes, the FTS5 virtual table supports standard SQL queries with the `MATCH` operator for full-text search. You can query the `src_fts` table directly using `SELECT * FROM src_fts WHERE src_fts MATCH 'functionName'`, though the application typically routes these through the BM25 ranking path for ordered results.

### Where is the tokenizer initialized during application startup?

The `cbm_camel_split` tokenizer is registered during the initialization phase in [`src/mcp/mcp.c`](https://github.com/DeusData/codebase-memory-mcp/blob/main/src/mcp/mcp.c) at line 367. The `register_camel_split_tokenizer` function calls the SQLite tokenizer registration API, making the custom tokenizer available before any indexing or search operations occur.