# How Needle Handles Off-Topic Input in Structured LLM Workflows

> Needle prevents off-topic input by returning empty function call arrays, guaranteeing predictable behavior when no tools match user intent in structured LLM workflows.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-09-12

---

**Needle refuses off-topic requests by returning an empty function call array instead of generating free-text responses, ensuring predictable behavior when no declared tools match the user intent.**

When building structured LLM applications with the cactus-compute/needle repository, controlling how the system responds to irrelevant queries is critical for production reliability. Unlike general-purpose chatbots that might hallucinate answers, Needle implements a strict refusal mechanism for off-topic input. This design ensures that the engine only responds when it can execute a declared tool, maintaining deterministic behavior across tool-calling and extraction workflows.

## The Core Refusal Contract

According to the official API documentation in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) at line 120, Needle defines off-topic input as any request that cannot be satisfied by the set of declared tools. When the engine encounters such input, it returns an empty function call list (`[]`) rather than attempting to generate free-text or hallucinated content. This empty array serves as an explicit, machine-readable signal that the query falls outside the defined tool schema.

## The Off-Topic Processing Pipeline

When the model receives an off-topic prompt, it executes a three-step validation and generation process:

### Tool Schema Verification

First, the system performs a **tool lookup** against the current tool schema(s). If none of the available tools match the user's intent, the request is immediately classified as off-topic. This validation step occurs before any response generation begins.

### Grammar-Constrained Empty Calls

Once classified as off-topic, the decoding grammar constrains the model to emit a call object with an empty `function_calls` field. As implemented in the engine, this guarantees syntactically correct JSON output while unambiguously signaling refusal. The constraint prevents the model from generating any alternative response format.

### Elimination of Free-Text Fallbacks

Because the grammar exclusively permits the empty call structure, the model **cannot** produce free-text answers, markdown explanations, or conversational filler. The response payload consists solely of the empty call object, ensuring that downstream applications can reliably detect refusals by checking for `function_calls: []`.

## Uniform Behavior Across APIs

The same refusal logic applies to the extraction API documented at line 20 of [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md). If the supplied extraction schema cannot match the input, the engine returns an empty call rather than attempting partial extraction or free-text explanation. This ensures a uniform contract across both tool-calling and extraction use-cases, simplifying error handling in client applications.

## Training for Refusal Behavior

### Representing Off-Topic Cases in Training Data

The repository's training pipeline uses specific examples to teach the model when to refuse. According to [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) at line 101, off-topic examples are represented using `"answers": []` in the training records. This empty answers array explicitly marks the sample as a refusal case that the model should learn to replicate.

### Special Handling During Fine-Tuning

The fine-tuning implementation in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) at line 57 treats these empty-answer records specially. Instead of penalizing the model for not generating content, the training process reinforces the behavior of producing empty function calls when no tool matches. This ensures that the learned policy respects the refusal contract defined in the API specification.

## Implementation Examples

The following examples demonstrate how off-topic handling works in practice using the Needle Python SDK.

First, declare a tool and attempt an off-topic query:

```python
import needle

@needle.tool
def add(a: int, b: int) -> dict:
    """Add two numbers and return the sum."""
    return {"result": a + b}

agent = needle.Needle(tools=[add])

# Off-topic query: asks about the weather, which no tool can handle

response = agent.run("What's the weather like in Paris today?")
print(response)

# Output:

# {'type': 'respond', 'function_calls': []}

```

The empty `function_calls` array confirms that Needle refused the off-topic request instead of generating a free-text weather report.

You can also observe this behavior using the low-level `complete()` method:

```python
resp = agent.complete("Tell me a joke about cats")
print(resp["function_calls"])   # []

```

## Key Source Files and Testing

Understanding the complete off-topic handling implementation requires examining these specific files:

- **[`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md)** – The Behavior section at line 120 defines the official contract that off-topic input maps to empty calls (`[]`), while line 20 establishes the same rule for extraction APIs.

- **[`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md)** – The Data format section at line 101 documents how off-topic training examples use `"answers": []` to represent refusal cases.

- **[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)** – Line 57 implements the special handling logic for off-topic records during the fine-tuning process.

- **[`tests/test_finetune.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_finetune.py)** – Contains unit tests verifying that the model correctly processes off-topic records with empty answer arrays.

- **[`needle/playground/server.py`](https://github.com/cactus-compute/needle/blob/main/needle/playground/server.py)** – Demonstrates runtime instantiation where agents may be configured with empty tool lists (`tools="[]"`), triggering the refusal behavior for all inputs.

## Summary

- Needle defines off-topic input as any query that does not match the declared tool schemas.
- The engine returns an empty `function_calls` array (`[]`) to signal refusal, never generating free-text responses.
- A constrained decoding grammar enforces this behavior by preventing any output format except the empty call object.
- Both the tool-calling and extraction APIs share this uniform refusal contract.
- Fine-tuning uses `"answers": []` in training data to teach the model when to refuse, with special handling in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py).

## Frequently Asked Questions

### What happens when Needle receives an off-topic query?

When Needle receives a query that does not match any declared tool schema, it returns a response object with an empty `function_calls` array. This empty array serves as an explicit refusal signal, indicating that no tool can satisfy the request. The response contains no free-text explanation or hallucinated content.

### Does Needle support a free-text fallback for off-topic inputs?

No. Needle does not support free-text fallbacks for off-topic queries. The decoding grammar is strictly constrained to emit only an empty call object when no tools match, ensuring that the model cannot generate markdown explanations, conversational responses, or fabricated answers.

### How is off-topic handling implemented during fine-tuning?

During fine-tuning, off-topic examples are marked with `"answers": []` in the training data, as documented in [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) at line 101. The fine-tuning code in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) at line 57 processes these records specially, reinforcing the model's behavior of returning empty function calls when it encounters inputs that lack matching tools.

### Is the off-topic behavior consistent across different Needle APIs?

Yes. Both the tool-calling API and the extraction API implement identical refusal behavior. According to [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md), if the supplied schema cannot match the input in either API, the engine returns an empty call, ensuring uniform error handling across different use-cases.