# How ds4 Handles Tool Calling and DSML Format Conversion: A Complete Technical Breakdown > Discover how ds4 manages tool calling and DSML format conversion using a three-stage pipeline for seamless integration with OpenAI-compatible APIs. Learn the technical details. - Repository: [Salvatore Sanfilippo/ds4](https://github.com/antirez/ds4) - Tags: deep-dive - Published: 2026-08-05 --- **ds4 uses a three-stage pipeline—DSML generation, streaming state-machine parsing, and JSON conversion—to bridge the model's native DSML markup with OpenAI-compatible tool-calling APIs.** The `antirez/ds4` repository implements a complete **tool-calling** pipeline that seamlessly converts between the model's native **DSML** (Domain-Specific Markup Language) syntax and standard JSON formats expected by client applications. This article examines the exact mechanisms, source file locations, and implementation details that make this bidirectional conversion possible. ## DSML Generation: Encoding Tool Calls in Native Syntax When the model decides to invoke a tool, `ds4` embeds the call inside a special DSML block using **`<｜DSML｜tool_calls>`** tags. The primary function responsible for this transformation is `append_dsml_tool_calls_text()` located in **[`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c)** (lines 2362–2382): ```c static void append_dsml_tool_calls_text(buf *b, const tool_calls *calls) { if (!calls || calls->len == 0) return; if (calls->raw_tool_text && calls->raw_tool_text[0]) { buf_puts(b, calls->raw_tool_text); return; } buf_puts(b, "\n\n<｜DSML｜tool_calls>\n"); for (int i = 0; i < calls->len; i++) { const tool_call *tc = &calls->v[i]; buf_puts(b, "<｜DSML｜invoke name=\""); append_dsml_attr_escaped(b, tc->name); buf_puts(b, "\">\n"); if (!append_dsml_arguments_from_json(b, tc->arguments, NULL)) { buf_puts(b, "<｜DSML｜parameter name=\"arguments\" string=\"true\">"); append_dsml_parameter_text(b, tc->arguments); buf_puts(b, "\n"); } buf_puts(b, "\n"); } buf_puts(b, ""); } ``` Key behaviors of this **DSML format conversion** process: - Empty tool call lists are silently skipped - Pre-serialized raw text can bypass re-encoding - Tool names are attribute-escaped via `append_dsml_attr_escaped()` - Arguments convert to nested **`<｜DSML｜parameter>`** tags when JSON conversion fails ## DSML Parsing: Streaming State-Machine Decoder During inference, `ds4` parses partially-generated DSML using a resilient streaming decoder. The parser tracks its position through enumerated states defined near **line 4570 in [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c)**: ```c enum { DSML_TOOL_BETWEEN_INVOKES, DSML_TOOL_BETWEEN_PARAMS, DSML_TOOL_PARAM_VALUE, DSML_TOOL_DONE, DSML_TOOL_ERROR, }; ``` The decoder operates in distinct parsing modes (lines 5831–5850): ```c enum { DSML_DECODE_OUTSIDE, DSML_DECODE_STRUCTURAL, DSML_DECODE_STRING_BODY, DSML_DECODE_JSON_STRUCTURAL, DSML_DECODE_JSON_STRING, }; ``` State transitions occur as the parser encounters tag boundaries. The following logic pattern appears throughout the decoder (approximately lines 6288–6440): ```c if (ts->state == DSML_TOOL_BETWEEN_INVOKES) { ts->state = DSML_TOOL_DONE; // after } else if (ts->state == DSML_TOOL_BETWEEN_PARAMS) { ts->state = DSML_TOOL_BETWEEN_INVOKES; // after } else if (ts->state == DSML_TOOL_PARAM_VALUE) { ts->state = DSML_TOOL_BETWEEN_PARAMS; // after } ``` This **streaming DSML parser** tolerates malformed or truncated output without crashing, extracting complete tool calls even from incomplete model responses. ## JSON Conversion: OpenAI-Compatible API Output After parsing, `ds4` serializes the collected `tool_calls` structure into standard JSON format using `append_tool_calls_json()` (lines 5353–5370): ```c static void append_tool_calls_json(buf *b, const tool_calls *calls, const char *id_prefix, const tool_schema_orders *orders) { // ... builds JSON array [...] } ``` The function integrates into the HTTP response builder at lines 7471–7472: ```c buf_puts(&b, ",\"tool_calls\":"); append_tool_calls_json(&b, calls, id, &r->tool_orders); ``` This produces **OpenAI-compatible tool-calling** JSON containing: - **`id`**: Prefixed identifiers (e.g., `"call_0"`) - **`type`**: Always `"function"` - **`function`**: Object with `name` and `arguments` ## Practical Implementation Examples ### Generating DSML from C ```c tool_calls my_calls = {0}; tool_calls_push(&my_calls, (tool_call){ .name = "search", .arguments = json_from_string("{\"query\":\"hello\"}") }); buf output = {0}; append_dsml_tool_calls_text(&output, &my_calls); ``` Output buffer contains: ``` <｜DSML｜tool_calls> <｜DSML｜invoke name="search"> <｜DSML｜parameter name="arguments" string="true">{"query":"hello"} ``` ### Parsing Raw Model Output ```c const char *raw = "<｜DSML｜tool_calls>…"; tool_calls parsed = {0}; dsml_tracker ts = {.state = DSML_TOOL_BETWEEN_INVOKES}; parse_dsml_block(&raw, &ts, &parsed); // populates parsed tool_calls ``` ### Converting to Client JSON ```c buf json_out = {0}; append_tool_calls_json(&json_out, &parsed, "call_", NULL); ``` Produces: ```json [ { "id": "call_0", "type": "function", "function": { "name": "search", "arguments": {"query": "hello"} } } ] ``` ## Source File Reference | File | Role in Tool-Calling Pipeline | |------|-------------------------------| | **[`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c)** | Core implementation: DSML generation (`append_dsml_tool_calls_text`), streaming parser (state machine at lines 4570+, 5831+, 6288+), JSON conversion (`append_tool_calls_json`) | | **[`ds4.h`](https://github.com/antirez/ds4/blob/main/ds4.h)** | `tool_calls` structure declarations, `DS4_LOG_TOOL` logging constant | | **[`tests/ds4_test.c`](https://github.com/antirez/ds4/blob/main/tests/ds4_test.c)** | Unit tests: `test_tool_call_quality()`, state-machine assertions | | **[`misc/COMPACT.md`](https://github.com/antirez/ds4/blob/main/misc/COMPACT.md)** | DSML syntax specification documentation | | **[`README.md`](https://github.com/antirez/ds4/blob/main/README.md)** | OpenAI-compatible API overview | ## Summary - **DSML generation** uses `append_dsml_tool_calls_text()` in [`ds4_server.c`](https://github.com/antirez/ds4/blob/main/ds4_server.c) to embed tool calls in model-native markup - **Streaming parsing** employs a five-state machine (`DSML_TOOL_BETWEEN_INVOKES` through `DSML_TOOL_ERROR`) with five decoder modes for resilient extraction - **JSON conversion** via `append_tool_calls_json()` produces OpenAI-compatible output for HTTP responses - The same `tool_calls` structure synchronizes both representations, ensuring consistency between DSML and JSON formats ## Frequently Asked Questions ### What is DSML in the ds4 context? DSML (Domain-Specific Markup Language) is a compact, XML-like syntax using vertical-bar delimiters (`<｜DSML｜tag>`) that the model generates natively for tool invocations. It provides a structured yet human-readable format for function calls with named parameters. ### How does ds4 handle incomplete or malformed DSML? The `ds4` streaming parser maintains explicit state (`DSML_TOOL_ERROR`) and continues processing rather than failing. It extracts any complete tool calls found before truncation or syntax errors, enabling graceful degradation during partial model outputs. ### Why does ds4 support both DSML and JSON formats? DSML serves as the model's native output format optimized for generation, while JSON provides the standard interface that OpenAI-compatible clients expect. The dual-format approach allows `ds4` to work with existing tooling without requiring client-side DSML parsers. ### Where are tool-calling events logged in ds4? Tool-related events use the `DS4_LOG_TOOL` logging type declared in [`ds4.h`](https://github.com/antirez/ds4/blob/main/ds4.h), enabling operators to filter and monitor tool invocation patterns separately from general inference logs.