# How the Heuristic Tool Parser in free-claude-code Detects and Converts Text Tool Calls

> Discover how the heuristic tool parser in free-claude-code detects text tool calls using the ● trigger and regex. Learn how it converts calls into Anthropic tool_use blocks.

- Repository: [Ali Khokhar/free-claude-code](https://github.com/Alishahryar1/free-claude-code)
- Tags: deep-dive
- Published: 2026-04-24

---

**The HeuristicToolParser in free-claude-code detects text-based tool calls by scanning for the `●` trigger character, then uses regex patterns to extract function names and parameter key-value pairs, converting them into Anthropic-compatible `tool_use` blocks.**

The free-claude-code repository provides a stateful streaming parser that bridges the gap between raw text outputs from OpenAI-compatible models and Anthropic's structured tool format. This heuristic tool parser processes streaming chunks in real-time, identifying tool invocations that appear as XML-like tags embedded in text and transforming them into the standardized JSON blocks required by the downstream SSE pipeline.

## The Architecture of the Heuristic Tool Parser

The core implementation resides in [`providers/common/heuristic_tool_parser.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/common/heuristic_tool_parser.py), where the `HeuristicToolParser` class maintains state across streaming chunks to reconstruct complete tool calls from fragmented input.

### State Machine Design

The parser operates through three mutually exclusive states defined by the `ParserState` enum:

- **TEXT** – Default state where the parser scans for the trigger character `●`
- **MATCHING_FUNCTION** – Active after detecting the trigger while searching for the `<function=Name>` pattern
- **PARSING_PARAMETERS** – Capturing `<parameter=key>value</parameter>` pairs until the tool call completes

State transitions occur inside the `feed()` method (lines 80-194) and `flush()` method (lines 198-225), with the parser buffering incomplete text until enough context exists to determine the next transition.

### Integration with OpenAICompatibleProvider

In [`providers/openai_compat.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/openai_compat.py), the `OpenAICompatibleProvider._stream_response_impl` method instantiates the parser and calls `heuristic_parser.feed(part.content)` for every incoming chunk. The method receives a tuple of filtered text and detected tools, emitting the text via `sse.emit_text_delta` while converting each tool dictionary into proper `tool_use` content blocks with unique IDs formatted as `toolu_heuristic_<8-hex>`.

## Detection Flow: From Raw Text to Structured Tools

The conversion process follows a deterministic pipeline that extracts structured data from unstructured text streams.

### Step 1: Trigger Detection

While in the `TEXT` state, the parser scans the buffer for the bullet character `●`. When found at line 81-88, the parser splits the buffer at the trigger index, appending everything before it to the filtered output and retaining the trigger for further processing. The state then transitions to `MATCHING_FUNCTION`.

### Step 2: Function Name Extraction

The parser applies the compiled regex `_FUNC_START_PATTERN = re.compile(r"●\s*<function=([^>]+)>")` at line 106. Upon matching, it captures the function name from the first group, generates a unique tool ID, and transitions to the `PARSING_PARAMETERS` state. If the pattern fails to match within a length guard, the parser falls back to treating the buffer as normal text.

### Step 3: Parameter Parsing

In the `PARSING_PARAMETERS` state, the parser repeatedly applies `_PARAM_PATTERN = re.compile(r"<parameter=([^>]+)>(.*?)(?:</parameter>|$)", re.DOTALL)` to extract key-value pairs. Each match updates `self._current_parameters` dictionary at line 46, storing the parameter name and value for eventual inclusion in the tool's input object.

### Step 4: Tool Finalization

The parser finalizes a tool call when any of three conditions occur at lines 157-171: detection of another `●` signaling a new tool, encountering non-tag text after having parsed at least one parameter, or reaching the end of the buffer. At lines 178-184, the parser constructs the Anthropic-compatible tool dictionary with the structure `{"type": "tool_use", "id": "...", "name": "...", "input": {...}}`, appends it to the detected tools list, and resets the state to `TEXT`.

## Control Token Sanitization and Edge Cases

Before any pattern matching occurs, the buffer undergoes sanitization through `_strip_control_tokens` (lines 44-48), which removes sentinel tokens like ` <|tool_call_end|> ` that leak from OpenAI backends. To handle streaming boundaries where control tokens might split across chunks, the parser uses `_split_incomplete_control_token_tail` (lines 49-65) to identify and preserve partial tokens in the buffer without emitting them as text.

When the stream ends, the provider calls `heuristic_parser.flush()` at lines 27-48 in the integration layer. This method captures any remaining parameter fragments even if closing `</parameter>` tags are missing, ensuring that incomplete tool calls are not lost during abrupt stream terminations.

## Implementation Examples

### Direct Parser API Usage

You can instantiate and use the parser directly for testing or custom implementations:

```python
from providers.common.heuristic_tool_parser import HeuristicToolParser

parser = HeuristicToolParser()

# Feed a complete tool call in one chunk

filtered, tools = parser.feed(
    "Analyzing codebase. ● <function=Grep>"
    "<parameter=pattern>TODO</parameter>"
    "<parameter=path>/src</parameter>"
)

print(filtered)  # → "Analyzing codebase. "

print(tools)     # → [{'type': 'tool_use', 'id': 'toolu_heuristic_a3f7b2c1',

                 #     'name': 'Grep',

                 #     'input': {'pattern': 'TODO', 'path': '/src'}}]

# Always flush when the stream ends to capture incomplete calls

remaining_tools = parser.flush()

```

### Integration in the SSE Pipeline

The standard usage within the provider implementation demonstrates the streaming pattern:

```python
from providers.common.heuristic_tool_parser import HeuristicToolParser

heuristic_parser = HeuristicToolParser()

# Inside OpenAICompatibleProvider._stream_response_impl

if delta.content:
    filtered_text, detected_tools = heuristic_parser.feed(delta.content)
    
    # Emit text deltas immediately

    if filtered_text:
        sse.emit_text_delta(filtered_text)
    
    # Convert detected tools to tool_use blocks

    for tool in detected_tools:
        sse.emit_tool_use_start(tool["id"], tool["name"])
        sse.emit_tool_use_input(json.dumps(tool["input"]))
        sse.emit_tool_use_stop(tool["id"])

```

The integration ensures that filtered text and tool calls interleave correctly in the Server-Sent Events stream, maintaining chronological order between narrative text and structured tool invocations.

## Summary

- **The `HeuristicToolParser` class in [`providers/common/heuristic_tool_parser.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/providers/common/heuristic_tool_parser.py) provides stateful streaming detection of text-based tool calls marked by the `●` character.**
- **Three parser states (TEXT, MATCHING_FUNCTION, PARSING_PARAMETERS) manage the transition from raw text to structured tool extraction.**
- **Regex patterns `_FUNC_START_PATTERN` and `_PARAM_PATTERN` extract function names and key-value pairs from XML-like tags.**
- **Control token stripping and split-token handling prevent OpenAI backend artifacts from corrupting the output.**
- **The `feed()` method returns filtered text and tool dictionaries, while `flush()` captures incomplete calls at stream end.**

## Frequently Asked Questions

### What format does the heuristic tool parser expect for text tool calls?

The parser expects tool calls to begin with the bullet character `●` followed by a function tag in the format `<function=FunctionName>`, then zero or more parameter tags in the format `<parameter=key>value</parameter>`. The function name is captured using the regex `r"●\s*<function=([^>]+)>"`, while parameters use `r"<parameter=([^>]+)>(.*?)(?:</parameter>|$)"` with the `re.DOTALL` flag to handle multi-line values.

### How does the parser handle incomplete or streaming tool calls?

The parser maintains a `_buffer` that accumulates text across multiple `feed()` calls. If a tool call splits across network chunks, the state machine remains in `MATCHING_FUNCTION` or `PARSING_PARAMETERS` until sufficient text arrives to complete the pattern matching. At stream termination, calling `flush()` extracts any partially captured parameters using `re.finditer(r"<parameter=([^>]+)>(.*)$", ...)` and emits the final tool dictionary.

### What happens to control tokens like ` <|tool_call_end|> ` during parsing?

Before pattern matching, the buffer passes through `_strip_control_tokens` which removes known sentinel tokens such as ` <|tool_call_end|> ` that OpenAI backends occasionally emit. Additionally, `_split_incomplete_control_token_tail` checks for partial control tokens at the end of chunks to prevent them from being erroneously emitted as text content while preserving the incomplete portion in the buffer for the next iteration.

### How does the parser maintain state across multiple chunks?

The parser uses an instance-level `ParserState` enum stored in `self._state` to track whether it is scanning for triggers, extracting function names, or accumulating parameters. Instance variables including `self._buffer`, `self._current_function_name`, `self._current_tool_id`, and `self._current_parameters` persist across `feed()` calls, enabling the reassembly of fragmented tool invocations that arrive in multiple SSE chunks.