How Needle Handles Off-Topic Input in Structured LLM Workflows

Needle refuses off-topic requests by returning an empty function call array instead of generating free-text responses, ensuring predictable behavior when no declared tools match the user intent.

When building structured LLM applications with the cactus-compute/needle repository, controlling how the system responds to irrelevant queries is critical for production reliability. Unlike general-purpose chatbots that might hallucinate answers, Needle implements a strict refusal mechanism for off-topic input. This design ensures that the engine only responds when it can execute a declared tool, maintaining deterministic behavior across tool-calling and extraction workflows.

The Core Refusal Contract

According to the official API documentation in doc/apis.md at line 120, Needle defines off-topic input as any request that cannot be satisfied by the set of declared tools. When the engine encounters such input, it returns an empty function call list ([]) rather than attempting to generate free-text or hallucinated content. This empty array serves as an explicit, machine-readable signal that the query falls outside the defined tool schema.

The Off-Topic Processing Pipeline

When the model receives an off-topic prompt, it executes a three-step validation and generation process:

Tool Schema Verification

First, the system performs a tool lookup against the current tool schema(s). If none of the available tools match the user's intent, the request is immediately classified as off-topic. This validation step occurs before any response generation begins.

Grammar-Constrained Empty Calls

Once classified as off-topic, the decoding grammar constrains the model to emit a call object with an empty function_calls field. As implemented in the engine, this guarantees syntactically correct JSON output while unambiguously signaling refusal. The constraint prevents the model from generating any alternative response format.

Elimination of Free-Text Fallbacks

Because the grammar exclusively permits the empty call structure, the model cannot produce free-text answers, markdown explanations, or conversational filler. The response payload consists solely of the empty call object, ensuring that downstream applications can reliably detect refusals by checking for function_calls: [].

Uniform Behavior Across APIs

The same refusal logic applies to the extraction API documented at line 20 of doc/apis.md. If the supplied extraction schema cannot match the input, the engine returns an empty call rather than attempting partial extraction or free-text explanation. This ensures a uniform contract across both tool-calling and extraction use-cases, simplifying error handling in client applications.

Training for Refusal Behavior

Representing Off-Topic Cases in Training Data

The repository's training pipeline uses specific examples to teach the model when to refuse. According to README.md at line 101, off-topic examples are represented using "answers": [] in the training records. This empty answers array explicitly marks the sample as a refusal case that the model should learn to replicate.

Special Handling During Fine-Tuning

The fine-tuning implementation in needle/model/finetune.py at line 57 treats these empty-answer records specially. Instead of penalizing the model for not generating content, the training process reinforces the behavior of producing empty function calls when no tool matches. This ensures that the learned policy respects the refusal contract defined in the API specification.

Implementation Examples

The following examples demonstrate how off-topic handling works in practice using the Needle Python SDK.

First, declare a tool and attempt an off-topic query:

import needle

@needle.tool
def add(a: int, b: int) -> dict:
    """Add two numbers and return the sum."""
    return {"result": a + b}

agent = needle.Needle(tools=[add])

# Off-topic query: asks about the weather, which no tool can handle

response = agent.run("What's the weather like in Paris today?")
print(response)

# Output:

# {'type': 'respond', 'function_calls': []}

The empty function_calls array confirms that Needle refused the off-topic request instead of generating a free-text weather report.

You can also observe this behavior using the low-level complete() method:

resp = agent.complete("Tell me a joke about cats")
print(resp["function_calls"])   # []

Key Source Files and Testing

Understanding the complete off-topic handling implementation requires examining these specific files:

• doc/apis.md – The Behavior section at line 120 defines the official contract that off-topic input maps to empty calls ([]), while line 20 establishes the same rule for extraction APIs.

• README.md – The Data format section at line 101 documents how off-topic training examples use "answers": [] to represent refusal cases.

• needle/model/finetune.py – Line 57 implements the special handling logic for off-topic records during the fine-tuning process.

• tests/test_finetune.py – Contains unit tests verifying that the model correctly processes off-topic records with empty answer arrays.

• needle/playground/server.py – Demonstrates runtime instantiation where agents may be configured with empty tool lists (tools="[]"), triggering the refusal behavior for all inputs.

Summary

• Needle defines off-topic input as any query that does not match the declared tool schemas.
• The engine returns an empty function_calls array ([]) to signal refusal, never generating free-text responses.
• A constrained decoding grammar enforces this behavior by preventing any output format except the empty call object.
• Both the tool-calling and extraction APIs share this uniform refusal contract.
• Fine-tuning uses "answers": [] in training data to teach the model when to refuse, with special handling in needle/model/finetune.py.

Frequently Asked Questions

What happens when Needle receives an off-topic query?

When Needle receives a query that does not match any declared tool schema, it returns a response object with an empty function_calls array. This empty array serves as an explicit refusal signal, indicating that no tool can satisfy the request. The response contains no free-text explanation or hallucinated content.

Does Needle support a free-text fallback for off-topic inputs?

No. Needle does not support free-text fallbacks for off-topic queries. The decoding grammar is strictly constrained to emit only an empty call object when no tools match, ensuring that the model cannot generate markdown explanations, conversational responses, or fabricated answers.

How is off-topic handling implemented during fine-tuning?

During fine-tuning, off-topic examples are marked with "answers": [] in the training data, as documented in README.md at line 101. The fine-tuning code in needle/model/finetune.py at line 57 processes these records specially, reinforcing the model's behavior of returning empty function calls when it encounters inputs that lack matching tools.

Is the off-topic behavior consistent across different Needle APIs?

Yes. Both the tool-calling API and the extraction API implement identical refusal behavior. According to doc/apis.md, if the supplied schema cannot match the input in either API, the engine returns an empty call, ensuring uniform error handling across different use-cases.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}