Can Needle 2 Be Used for Tasks Other Than Tool Calling? Exploring the Full LLM Stack

Yes, Needle 2 is a complete LLM inference and fine-tuning framework that functions as a vanilla text generator when no tools are provided, supporting batch inference, fine-tuning, synthetic data generation, and cross-platform deployment beyond its optional tool-calling capabilities.

Needle 2, developed by Cactus Compute, is architected as a full-featured inference stack rather than a single-purpose tool wrapper. While it offers robust function-calling capabilities through the @tool decorator, its core JAX-based transformer engine operates independently of the tool layer, enabling standard text generation, batch processing, and model customization workflows.

Core Architecture: Three Decoupled Layers

The repository cactus-compute/needle organizes functionality into loosely coupled layers. Understanding this separation clarifies why tool calling remains strictly optional.

Tokenization and Prompt Encoding

The tokenizer handles special markers such as <tools>…</tools> and <tool_call>…</tool_call>, but these are only inserted when a tool schema is explicitly supplied. In needle/model/tokenizer.py, the marker definitions (lines 16–21) are conditionally applied based on the presence of tool configurations. When instantiated without tools, the tokenizer skips these markers entirely, processing prompts as standard text sequences.

Model Inference Engine

The heart of Needle 2 resides in needle/model/run.py, which implements the JAX-based transformer forward pass. The generate function (lines 176–212) and batch_generate function (lines 18–34) handle token sampling, constrained decoding, and probability calculations regardless of whether tools are active. This engine loads checkpoints, manages KV caching, and returns raw token streams or structured statistics without requiring tool definitions.

Optional Agent and Tool Layer

Located in needle/agent/tools.py, the @tool decorator (lines 63–66) and build_schema utility provide a lightweight runtime for executing Python functions when the model emits tool requests. This layer is completely optional—omitting the tools parameter during Needle instantiation bypasses the schema injection and execution loop, causing the model to behave exactly like a standard decoder-only LLM.

Practical Use Cases Beyond Tool Calling

Needle 2 exposes multiple sub-commands through needle/cli.py (lines 10–38 and 82–106) that demonstrate its versatility as a general-purpose inference framework.

Plain Text Generation

When initialized without tools, Needle() defaults to free-form text generation. The same generate method called with tools=[] or the default empty list executes standard autoregressive decoding without special marker insertion or post-processing.

Batch Inference with Token-Level Signals

For evaluation and research workflows, batch_generate supports return_signals=True to output per-token log probabilities, entropies, and completion lengths. This feature, implemented in needle/model/run.py (lines 18–34), enables statistical analysis of model confidence across datasets without any tool-related overhead.

Fine-Tuning on Custom Data

The framework includes a complete training loop via finetune_local in needle/model/finetune.py (lines 24–38). Users can apply LoRA adapters to base checkpoints, train on JSONL-formatted datasets, and export quantized models—all independently of the tool-calling pipeline.

Synthetic Data Generation

needle generate-data leverages OpenRouter to create training examples for specific tool schemas, but the underlying generation mechanism (lines 41–49 in finetune.py) utilizes the standard inference engine. This capability can be adapted for general prompt-completion pair generation by modifying the input schema.

Cross-Platform Deployment

needle/agent/fetch.py (lines 88–101) handles platform-specific library extraction, downloading native shared libraries (libneedle.so or equivalent) for macOS, Linux, Windows, Android, and iOS. This infrastructure supports edge deployment of fine-tuned models for text generation tasks without requiring tool execution environments.

How to Use Needle 2 Without Tool Calling

The following examples demonstrate standard LLM workflows using the Python API and CLI, bypassing the optional tool layer entirely.

Basic Text Generation

Instantiate Needle without tools to perform vanilla generation:

from needle import Needle

agent = Needle()
text = agent.generate(
    prompt="Explain the significance of transformer architectures in modern NLP.",
    max_new_tokens=256,
    temperature=0.8,
    stream=False,
)
print(text)

This invokes needle.model.run.generate (lines 176–212), executing the transformer forward pass without tool marker injection.

Batch Processing with Statistics

Process multiple prompts efficiently while extracting generation metadata:

agent = Needle()
outputs = agent.batch_generate(
    prompts=[
        "Summarize the theory of relativity.",
        "Describe the water cycle in simple terms."
    ],
    max_new_tokens=128,
    return_signals=True,
)

for output in outputs:
    print(f"Text: {output['text']}")
    print(f"Mean log-probability: {output['mean_logprob']}")

The batch_generate implementation gathers per-token statistics (lines 18–34 in run.py) useful for uncertainty quantification and filtering low-confidence generations.

Fine-Tuning a Base Model

Train a LoRA adapter on custom data using the CLI:

needle finetune training_data.jsonl \
  --checkpoint cactus-compute/needle2 \
  --epochs 3 \
  --lora-rank 16 \
  --bits 4 \
  --out domain_adapter.pkl

This executes finetune_local (lines 24–38 in finetune.py), updating adapter weights while keeping the base model frozen.

Platform-Specific Engine Fetching

Deploy to ARM64 Linux servers by fetching the appropriate native library:

needle fetch --platform-tag manylinux2014_aarch64 --out ./needle_libs

The fetch.fetch_library function (lines 88–101 in fetch.py) extracts the compatible shared library from Hugging Face repositories, enabling optimized inference on target hardware.

Interactive Web Interface

Launch the playground UI for manual testing and prompt engineering:

needle playground --port 7860 --checkpoint ./my_model.cact

This starts the Flask-based server defined in needle/playground/server.py, providing a browser interface for text generation experimentation.

Summary

• Needle 2 operates as a standard LLM when the tools parameter is omitted, using the same generate and batch_generate functions for vanilla text completion.
• The tokenizer in needle/model/tokenizer.py conditionally inserts tool markers; without tools, prompts remain unmodified standard text.
• Batch inference supports statistical signal extraction (log probabilities, entropies) through return_signals=True, useful for research and filtering applications.
• Fine-tuning workflows via needle finetune and the finetune_local function enable domain adaptation without involving tool-calling logic.
• Cross-platform deployment through needle fetch supports edge inference on diverse hardware architectures.
• All functionality—generation, training, data synthesis, and deployment—shares the core JAX inference engine, making tool calling purely optional schema-driven syntactic sugar.

Frequently Asked Questions

Can Needle 2 generate text without any function definitions?

Yes. Instantiating Needle() without passing a tools argument defaults to standard autoregressive text generation. The tokenizer omits <tools> markers, and the generation loop in needle/model/run.py produces free-form completions exactly like any other decoder-only model.

Does batch generation work differently when tools are disabled?

No. The batch_generate method in needle/model/run.py (lines 18–34) uses identical code paths regardless of tool configuration. When tools are absent, the function simply skips the parsing logic for <tool_call> blocks and returns raw text completions, optionally including per-token probability statistics via the return_signals parameter.

Is it possible to fine-tune Needle 2 for general text completion rather than tool use?

Absolutely. The needle finetune CLI command and underlying finetune_local function (lines 24–38 in needle/model/finetune.py) operate on standard JSONL-formatted prompt-completion pairs. These utilities apply LoRA adapters to the base checkpoint without requiring tool schemas, making the framework suitable for domain-specific text generation tasks.

How does Needle 2 handle deployment on mobile or edge devices without tool execution capabilities?

The needle fetch command downloads platform-specific native libraries (lines 88–101 in needle/agent/fetch.py) containing the optimized JAX inference engine. Once fetched, the Needle class loads these libraries to run pure text generation locally on macOS, Linux, Windows, Android, or iOS hardware, completely independent of the Python-based tool runtime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}