How the Heuristic Tool Parser in free-claude-code Detects and Converts Text Tool Calls

The HeuristicToolParser in free-claude-code detects text-based tool calls by scanning for the ● trigger character, then uses regex patterns to extract function names and parameter key-value pairs, converting them into Anthropic-compatible tool_use blocks.

The free-claude-code repository provides a stateful streaming parser that bridges the gap between raw text outputs from OpenAI-compatible models and Anthropic's structured tool format. This heuristic tool parser processes streaming chunks in real-time, identifying tool invocations that appear as XML-like tags embedded in text and transforming them into the standardized JSON blocks required by the downstream SSE pipeline.

The Architecture of the Heuristic Tool Parser

The core implementation resides in providers/common/heuristic_tool_parser.py, where the HeuristicToolParser class maintains state across streaming chunks to reconstruct complete tool calls from fragmented input.

State Machine Design

The parser operates through three mutually exclusive states defined by the ParserState enum:

• TEXT – Default state where the parser scans for the trigger character ●
• MATCHING_FUNCTION – Active after detecting the trigger while searching for the <function=Name> pattern
• PARSING_PARAMETERS – Capturing <parameter=key>value</parameter> pairs until the tool call completes

State transitions occur inside the feed() method (lines 80-194) and flush() method (lines 198-225), with the parser buffering incomplete text until enough context exists to determine the next transition.

Integration with OpenAICompatibleProvider

In providers/openai_compat.py, the OpenAICompatibleProvider._stream_response_impl method instantiates the parser and calls heuristic_parser.feed(part.content) for every incoming chunk. The method receives a tuple of filtered text and detected tools, emitting the text via sse.emit_text_delta while converting each tool dictionary into proper tool_use content blocks with unique IDs formatted as toolu_heuristic_<8-hex>.

Detection Flow: From Raw Text to Structured Tools

The conversion process follows a deterministic pipeline that extracts structured data from unstructured text streams.

Step 1: Trigger Detection

While in the TEXT state, the parser scans the buffer for the bullet character ●. When found at line 81-88, the parser splits the buffer at the trigger index, appending everything before it to the filtered output and retaining the trigger for further processing. The state then transitions to MATCHING_FUNCTION.

Step 2: Function Name Extraction

The parser applies the compiled regex _FUNC_START_PATTERN = re.compile(r"●\s*<function=([^>]+)>") at line 106. Upon matching, it captures the function name from the first group, generates a unique tool ID, and transitions to the PARSING_PARAMETERS state. If the pattern fails to match within a length guard, the parser falls back to treating the buffer as normal text.

Step 3: Parameter Parsing

In the PARSING_PARAMETERS state, the parser repeatedly applies _PARAM_PATTERN = re.compile(r"<parameter=([^>]+)>(.*?)(?:</parameter>|$)", re.DOTALL) to extract key-value pairs. Each match updates self._current_parameters dictionary at line 46, storing the parameter name and value for eventual inclusion in the tool's input object.

Step 4: Tool Finalization

The parser finalizes a tool call when any of three conditions occur at lines 157-171: detection of another ● signaling a new tool, encountering non-tag text after having parsed at least one parameter, or reaching the end of the buffer. At lines 178-184, the parser constructs the Anthropic-compatible tool dictionary with the structure {"type": "tool_use", "id": "...", "name": "...", "input": {...}}, appends it to the detected tools list, and resets the state to TEXT.

Control Token Sanitization and Edge Cases

Before any pattern matching occurs, the buffer undergoes sanitization through _strip_control_tokens (lines 44-48), which removes sentinel tokens like <|tool_call_end|> that leak from OpenAI backends. To handle streaming boundaries where control tokens might split across chunks, the parser uses _split_incomplete_control_token_tail (lines 49-65) to identify and preserve partial tokens in the buffer without emitting them as text.

When the stream ends, the provider calls heuristic_parser.flush() at lines 27-48 in the integration layer. This method captures any remaining parameter fragments even if closing </parameter> tags are missing, ensuring that incomplete tool calls are not lost during abrupt stream terminations.

Implementation Examples

Direct Parser API Usage

You can instantiate and use the parser directly for testing or custom implementations:

from providers.common.heuristic_tool_parser import HeuristicToolParser

parser = HeuristicToolParser()

# Feed a complete tool call in one chunk

filtered, tools = parser.feed(
    "Analyzing codebase. ● <function=Grep>"
    "<parameter=pattern>TODO</parameter>"
    "<parameter=path>/src</parameter>"
)

print(filtered)  # → "Analyzing codebase. "

print(tools)     # → [{'type': 'tool_use', 'id': 'toolu_heuristic_a3f7b2c1',

                 #     'name': 'Grep',

                 #     'input': {'pattern': 'TODO', 'path': '/src'}}]

# Always flush when the stream ends to capture incomplete calls

remaining_tools = parser.flush()

Integration in the SSE Pipeline

The standard usage within the provider implementation demonstrates the streaming pattern:

from providers.common.heuristic_tool_parser import HeuristicToolParser

heuristic_parser = HeuristicToolParser()

# Inside OpenAICompatibleProvider._stream_response_impl

if delta.content:
    filtered_text, detected_tools = heuristic_parser.feed(delta.content)
    
    # Emit text deltas immediately

    if filtered_text:
        sse.emit_text_delta(filtered_text)
    
    # Convert detected tools to tool_use blocks

    for tool in detected_tools:
        sse.emit_tool_use_start(tool["id"], tool["name"])
        sse.emit_tool_use_input(json.dumps(tool["input"]))
        sse.emit_tool_use_stop(tool["id"])

The integration ensures that filtered text and tool calls interleave correctly in the Server-Sent Events stream, maintaining chronological order between narrative text and structured tool invocations.

Summary

• The HeuristicToolParser class in providers/common/heuristic_tool_parser.py provides stateful streaming detection of text-based tool calls marked by the ● character.
• Three parser states (TEXT, MATCHING_FUNCTION, PARSING_PARAMETERS) manage the transition from raw text to structured tool extraction.
• Regex patterns _FUNC_START_PATTERN and _PARAM_PATTERN extract function names and key-value pairs from XML-like tags.
• Control token stripping and split-token handling prevent OpenAI backend artifacts from corrupting the output.
• The feed() method returns filtered text and tool dictionaries, while flush() captures incomplete calls at stream end.

Frequently Asked Questions

What format does the heuristic tool parser expect for text tool calls?

The parser expects tool calls to begin with the bullet character ● followed by a function tag in the format <function=FunctionName>, then zero or more parameter tags in the format <parameter=key>value</parameter>. The function name is captured using the regex r"●\s*<function=([^>]+)>", while parameters use r"<parameter=([^>]+)>(.*?)(?:</parameter>|$)" with the re.DOTALL flag to handle multi-line values.

How does the parser handle incomplete or streaming tool calls?

The parser maintains a _buffer that accumulates text across multiple feed() calls. If a tool call splits across network chunks, the state machine remains in MATCHING_FUNCTION or PARSING_PARAMETERS until sufficient text arrives to complete the pattern matching. At stream termination, calling flush() extracts any partially captured parameters using re.finditer(r"<parameter=([^>]+)>(.*)$", ...) and emits the final tool dictionary.

What happens to control tokens like <|tool_call_end|> during parsing?

Before pattern matching, the buffer passes through _strip_control_tokens which removes known sentinel tokens such as <|tool_call_end|> that OpenAI backends occasionally emit. Additionally, _split_incomplete_control_token_tail checks for partial control tokens at the end of chunks to prevent them from being erroneously emitted as text content while preserving the incomplete portion in the buffer for the next iteration.

How does the parser maintain state across multiple chunks?

The parser uses an instance-level ParserState enum stored in self._state to track whether it is scanning for triggers, extracting function names, or accumulating parameters. Instance variables including self._buffer, self._current_function_name, self._current_tool_id, and self._current_parameters persist across feed() calls, enabling the reassembly of fragmented tool invocations that arrive in multiple SSE chunks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}