# Needle 2 FFI Functions: Complete Reference for the Inference Engine C Interface

> Discover the 8 core FFI functions for the Needle 2 inference engine. Load models, set parameters, infer, generate, tokenize, and detokenize without a Python runtime.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: api-reference
- Published: 2026-08-30

---

**The Needle 2 inference engine exposes eight core C FFI functions—`needle_load_model`, `needle_unload_model`, `needle_set_params`, `needle_infer`, `needle_generate`, `needle_tokenize`, `needle_detokenize`, and `needle_get_version`—defined in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) to enable zero-overhead inference from C, C++, Rust, and other languages without requiring a Python runtime.**

The cactus-compute/needle repository provides a high-performance inference engine built on GGML/GGUF quantization. For system-level integrations, the Needle 2 FFI functions wrap the engine's Python internals via CFFI bindings, allowing external programs to load quantized checkpoints, manage tokenization, and execute generation loops through a stable C Application Binary Interface (ABI).

## Core FFI Function Reference

All foreign function interface definitions reside in **[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)**, which exports a C-compatible shared library (`libneedle_ffi.so` on Linux, `needle_ffi.dll` on Windows, or `libneedle_ffi.dylib` on macOS). The following table documents the complete public API surface:

| Function | Purpose | C Signature |
|----------|---------|-------------|
| **`needle_load_model`** | Loads a quantized GGML/GGUF checkpoint and tokenizer into memory, returning an opaque handle. | `void* needle_load_model(const char* checkpoint_path, const char* tokenizer_path, const char* device)` |
| **`needle_unload_model`** | Releases all resources associated with a model handle. | `void needle_unload_model(void* model_handle)` |
| **`needle_set_params`** | Configures generation parameters (temperature, top-p, max tokens) for a specific model instance. | `int needle_set_params(void* model_handle, const needle_params_t* params)` |
| **`needle_infer`** | Executes a single forward pass on tokenized input and returns the next token ID. | `int needle_infer(void* model_handle, const int32_t* input_ids, size_t input_len, int32_t* output_id)` |
| **`needle_generate`** | Performs autoregressive generation for a complete sequence, writing token IDs to an output buffer. | `int needle_generate(void* model_handle, const int32_t* prompt_ids, size_t prompt_len, int32_t* out_ids, size_t* out_len)` |
| **`needle_tokenize`** | Converts UTF-8 text into integer token IDs using the model's bundled SentencePiece or BPE tokenizer. | `int needle_tokenize(const char* text, int32_t* token_buf, size_t buf_len, size_t* token_len)` |
| **`needle_detokenize`** | Converts an array of token IDs back into a null-terminated UTF-8 string. | `int needle_detokenize(const int32_t* tokens, size_t token_len, char* out_str, size_t out_buf_len)` |
| **`needle_get_version`** | Returns the library version string (e.g., `"2.0.11"`). | `const char* needle_get_version(void)` |

All functions return integer status codes where `0` indicates success and non-zero values signal errors (invalid handle, buffer overflow, or generation failure).

## Implementation Architecture

The FFI layer in **[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)** serves as a thin translation shim between C types and the high-level Python inference logic. When you invoke `needle_load_model`, the wrapper:

1. Validates input paths and device strings ("cpu", "cuda", "metal").
2. Instantiates the `Model` class defined in **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)**, which handles GGML tensor allocation and KV-cache management.
3. Loads the tokenizer implementation from **[`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py)**.
4. Returns an opaque `void*` pointer to the Python object, cast for C compatibility.

The generation functions (`needle_generate` and `needle_infer`) delegate to the sampling and forward-pass methods in **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)**, while tokenization calls utilize the encoding/decoding logic maintained in **[`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py)**. This architecture ensures that the C interface automatically benefits from optimizations in the underlying GGML compute graph and quantization kernels defined in **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)**.

## Integration Examples

### C Application Integration

The following complete example demonstrates loading a quantized model, tokenizing a prompt, generating a response, and detokenizing the output. The code assumes you have generated the [`needle_ffi.h`](https://github.com/cactus-compute/needle/blob/main/needle_ffi.h) header during the Python package installation (`pip install needle`).

```c
#include <stdio.h>
#include <stdint.h>
#include "needle_ffi.h"

int main(void) {
    // Initialize model with CPU backend (use "cuda" or "metal" for GPU acceleration)
    void *model = needle_load_model(
        "models/ggml-model-q8_0.bin",
        "tokenizer.model",
        "cpu"
    );
    if (!model) {
        fprintf(stderr, "Failed to load model checkpoint\n");
        return 1;
    }

    // Tokenize input prompt
    const char *prompt = "Explain quantum computing in one sentence.";
    int32_t prompt_ids[256];
    size_t prompt_len = 0;
    
    if (needle_tokenize(prompt, prompt_ids, 256, &prompt_len) != 0) {
        fprintf(stderr, "Tokenization failed\n");
        needle_unload_model(model);
        return 1;
    }

    // Generate up to 64 new tokens
    int32_t output_ids[320];
    size_t output_len = 0;
    
    if (needle_generate(model, prompt_ids, prompt_len, 
                       output_ids, &output_len) != 0) {
        fprintf(stderr, "Generation failed\n");
        needle_unload_model(model);
        return 1;
    }

    // Convert token IDs back to text
    char result[1024];
    if (needle_detokenize(output_ids, output_len, 
                         result, sizeof(result)) != 0) {
        fprintf(stderr, "Detokenization failed\n");
        needle_unload_model(model);
        return 1;
    }

    printf("Generated text: %s\n", result);
    
    // Cleanup
    needle_unload_model(model);
    return 0;
}

```

**Key implementation notes:**

- **Memory ownership**: The caller manages all buffer allocations; `needle_load_model` returns the only heap-allocated object that the caller does not own, which must be freed via `needle_unload_model`.
- **Device selection**: Pass `"cpu"` for GGML CPU inference, `"cuda"` for NVIDIA GPUs, or `"metal"` for Apple Silicon.
- **Error handling**: Always check return codes; `needle_generate` returns non-zero if the requested `out_len` exceeds available buffer space.

### Rust Dynamic Loading

For Rust applications requiring runtime linking (avoiding static compilation against the FFI), use the `libloading` crate to resolve symbols from the shared library. This pattern matches how the Python package distributes its compiled artifacts.

```rust
use libloading::{Library, Symbol};
use std::ffi::{CStr, CString};
use std::os::raw::{c_char, c_int, c_void};

type LoadModel = unsafe extern "C" fn(*const c_char, *const c_char, *const c_char) -> *mut c_void;
type UnloadModel = unsafe extern "C" fn(*mut c_void);
type Tokenize = unsafe extern "C" fn(*const c_char, *mut i32, usize, *mut usize) -> c_int;
type Generate = unsafe extern "C" fn(*mut c_void, *const i32, usize, *mut i32, *mut usize) -> c_int;
type Detokenize = unsafe extern "C" fn(*const i32, usize, *mut c_char, usize) -> c_int;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let lib = Library::new("libneedle_ffi.so")?;

    unsafe {
        let load: Symbol<LoadModel> = lib.get(b"needle_load_model")?;
        let unload: Symbol<UnloadModel> = lib.get(b"needle_unload_model")?;
        let tokenize: Symbol<Tokenize> = lib.get(b"needle_tokenize")?;
        let generate: Symbol<Generate> = lib.get(b"needle_generate")?;
        let detokenize: Symbol<Detokenize> = lib.get(b"needle_detokenize")?;

        // Load model
        let model = load(
            CString::new("models/ggml-model-q4_0.bin")?.as_ptr(),
            CString::new("tokenizer.json")?.as_ptr(),
            CString::new("cpu")?.as_ptr(),
        );
        assert!(!model.is_null());

        // Tokenize
        let prompt = CString::new("Write a haiku about rust.")?;
        let mut ids = [0i32; 128];
        let mut len: usize = 0;
        assert_eq!(0, tokenize(prompt.as_ptr(), ids.as_mut_ptr(), ids.len(), &mut len as *mut usize));

        // Generate
        let mut out = [0i32; 256];
        let mut out_len: usize = 0;
        assert_eq!(0, generate(model, ids.as_ptr(), len, out.as_mut_ptr(), &mut out_len as *mut usize));

        // Detokenize
        let mut buf = [0u8; 1024];
        assert_eq!(0, detokenize(out.as_ptr(), out_len, buf.as_mut_ptr() as *mut c_char, buf.len()));
        
        println!("Output: {}", CStr::from_ptr(buf.as_ptr() as *const c_char).to_string_lossy());
        
        unload(model);
    }

    Ok(())
}

```

**Key implementation notes:**

- **Type safety**: Define `unsafe extern "C"` function pointers matching the signatures exported from [`export.py`](https://github.com/cactus-compute/needle/blob/main/export.py).
- **String handling**: Convert Rust `String` types to `CString` to ensure null-termination for the C ABI boundary.
- **Resource cleanup**: Call `needle_unload_model` before dropping the `Library` handle to prevent dangling pointers.

## Summary

- **Eight core functions** comprise the complete Needle 2 FFI surface: model lifecycle management (`needle_load_model`, `needle_unload_model`), generation control (`needle_set_params`, `needle_infer`, `needle_generate`), text processing (`needle_tokenize`, `needle_detokenize`), and metadata (`needle_get_version`).
- **Source location**: All bindings are implemented in **[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)**, which wraps the Python inference logic found in **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)** and tokenization utilities in **[`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py)**.
- **Library naming**: Link against `libneedle_ffi.so` (Linux), `needle_ffi.dll` (Windows), or `libneedle_ffi.dylib` (macOS) generated during the Python package build.
- **Memory model**: The FFI uses a handle-based design where `void* model_handle` represents the Python Model instance; all other buffers are caller-allocated.
- **Format support**: The interface specifically handles **GGML** and **GGUF** quantized checkpoints, with automatic device selection for CPU, CUDA, and Metal backends.

## Frequently Asked Questions

### Which shared library file do I need to link against?

Link against `libneedle_ffi.so` on Linux systems, `needle_ffi.dll` on Windows, or `libneedle_ffi.dylib` on macOS. These files are compiled automatically when you install the `needle` package via pip, typically located in the package's installation directory under `needle/lib/`.

### Do I need to install Python to use the Needle 2 FFI functions at runtime?

No. While the shared library is generated from Python source code using CFFI, the resulting binary is a standalone shared object with no runtime dependency on `libpython` or the Python interpreter. You only need Python during the build/installation phase to generate the library file.

### What model formats are supported by `needle_load_model`?

The function accepts **GGML** and **GGUF** quantized checkpoint files produced by the GGML conversion tools. You must also provide the corresponding tokenizer file (SentencePiece `.model` or HuggingFace [`tokenizer.json`](https://github.com/cactus-compute/needle/blob/main/tokenizer.json)) as the second argument. The implementation in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) validates the file headers before passing them to the GGML loader in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py).

### How does memory management work for the opaque `model_handle`?

The `void*` pointer returned by `needle_load_model` references a Python Model object retained in memory by the CFFI layer. You must explicitly call `needle_unload_model` to decrement the reference count and free associated GGML tensors, KV caches, and tokenizer objects. Failure to call this function results in memory leaks. All other buffers (token arrays, output strings) are allocated and freed by the caller.