How the Function Calling Strategy Handles Streaming vs Blocking LLM Responses

The Function Calling strategy checks for the STREAM_TOOL_CALL model feature to determine whether to process LLM outputs as a streaming generator (yielding tokens incrementally) or a blocking response (processing the complete result at once), handling tool call extraction accordingly in strategies/function_calling.py.

The junjiem/dify-plugin-agent-mcp_sse repository implements a Function Calling agent strategy that dynamically adapts to different LLM response modes. This capability allows the agent to optimize for either low-latency streaming outputs or complete blocking responses when executing tool calls. Understanding how the strategy handles these two modes is essential for developers configuring MCP (Model Context Protocol) agents in Dify workflows.

Detecting the Response Mode with STREAM_TOOL_CALL

The strategy determines the response mode by inspecting the model's declared capabilities before invoking the LLM. In strategies/function_calling.py, the code checks for the ModelFeature.STREAM_TOOL_CALL feature flag at lines 106-110.

If the model supports streaming tool calls, the stream parameter is set to True. Otherwise, the strategy defaults to blocking mode with stream set to False. This boolean flag is then passed to the LLM invocation method at lines 167-175, determining whether the underlying model client returns a generator or a complete result object.

Processing Streaming LLM Responses

When stream is enabled, the LLM returns a generator yielding LLMResultChunk objects. The strategy processes these incrementally in the _invoke method of FunctionCallingAgentStrategy.

For each chunk, the strategy performs the following actions:

• Tool call detection: Uses self.check_tool_calls(chunk) at lines 71-75 to identify if the current chunk contains function call signatures.
• Tool call extraction: Invokes self.extract_tool_calls(chunk) at lines 88-94 to parse the tool call parameters from the streaming delta.
• Incremental text delivery: Appends the textual delta to the accumulating response and yields intermediate text messages immediately using yield self.create_text_message() at lines 98-106, providing real-time feedback to the user.

This approach minimizes latency by emitting tokens as they arrive while simultaneously monitoring for tool invocations that may interrupt the text generation.

Processing Blocking LLM Responses

When streaming is not supported, the LLM returns a complete LLMResult object containing the full response. The strategy handles this in the blocking branch of the _invoke method.

The blocking processing flow involves:

• Complete result analysis: Receives the full LLMResult object instead of a generator.
• Tool call detection: Calls self.check_blocking_tool_calls(result) at lines 77-81 to scan the entire response for function calls.
• Tool call extraction: Uses self.extract_blocking_tool_calls(result) at lines 84-90 to retrieve all tool call parameters from the complete message.
• Single message emission: Yields the entire textual response in one operation at lines 124-132 using yield self.create_text_message(), ensuring atomic delivery of the complete thought.

This mode simplifies tool call handling by processing the entire context at once but sacrifices the progressive rendering benefits of streaming.

Implementation Details and Code Structure

The dual-mode handling is implemented in strategies/function_calling.py within the FunctionCallingAgentStrategy class. The core logic resides in the _invoke method, which branches based on whether chunks is a generator or a complete result.

Streaming detection logic (lines 106-110):

if ModelFeature.STREAM_TOOL_CALL in features:
    stream = True
else:
    stream = False

LLM invocation with mode flag (lines 167-175):

chunks = self.session.model.llm.invoke(
    model_configured,
    prompt_messages,
    stream=stream,
    **invoke_params
)

Streaming processing branch (simplified from lines 88-106):

if isinstance(chunks, Generator):
    for chunk in chunks:
        if self.check_tool_calls(chunk):
            tool_calls.extend(self.extract_tool_calls(chunk))
        if chunk.delta.message.content:
            yield self.create_text_message(str(chunk.delta.message.content))

Blocking processing branch (simplified from lines 77-90, 124-132):

else:
    result = cast(LLMResult, chunks)
    if self.check_blocking_tool_calls(result):
        tool_calls.extend(self.extract_blocking_tool_calls(result))
    if result.message.content:
        yield self.create_text_message(str(result.message.content))

Supporting files include strategies/function_calling.yaml for declarative configuration, utils/mcp_client.py for MCP server interactions, and strategies/base.py for shared history management utilities.

Summary

The Function Calling strategy in junjiem/dify-plugin-agent-mcp_sse implements a sophisticated dual-mode architecture for handling LLM responses:

• Capability-based detection: Automatically selects streaming or blocking mode by checking for the STREAM_TOOL_CALL model feature before invocation.
• Real-time streaming: Processes LLMResultChunk objects incrementally, yielding text tokens immediately while extracting tool calls from partial deltas.
• Atomic blocking: Handles complete LLMResult objects, scanning the full response for tool calls before emitting the entire message.
• Unified tool handling: Both paths converge on the same tool execution logic after response processing, ensuring consistent behavior regardless of transmission mode.

This design allows the agent to optimize for latency with compatible models while maintaining compatibility with blocking-only LLM providers.

Frequently Asked Questions

How does the strategy determine whether to use streaming or blocking mode?

The strategy inspects the model's feature list for ModelFeature.STREAM_TOOL_CALL at lines 106-110 in strategies/function_calling.py. If the feature is present, it sets stream=True for the LLM invocation; otherwise, it defaults to stream=False for blocking mode.

What happens to tool calls when using streaming responses?

During streaming, the strategy processes each LLMResultChunk incrementally. It calls check_tool_calls() and extract_tool_calls() on every chunk to detect function signatures as they arrive. Tool calls are accumulated across chunks, and text deltas are yielded immediately unless a tool call is pending.

Can the Function Calling strategy handle models that don't support streaming tool calls?

Yes. The strategy explicitly supports blocking mode for models lacking the STREAM_TOOL_CALL feature. In this mode, it receives a complete LLMResult object and uses check_blocking_tool_calls() and extract_blocking_tool_calls() to parse all tool calls from the finished response before yielding the full text.

Where is the core logic for switching between streaming and blocking implemented?

The dual-mode logic resides in the _invoke method of the FunctionCallingAgentStrategy class within strategies/function_calling.py. Lines 88-132 contain the branching logic that determines whether to iterate over a generator (streaming) or cast to LLMResult (blocking), with specific helper methods handling each mode's tool call extraction requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}