Needle 2 FFI Functions: Complete Reference for the Inference Engine C Interface

The Needle 2 inference engine exposes eight core C FFI functions—needle_load_model, needle_unload_model, needle_set_params, needle_infer, needle_generate, needle_tokenize, needle_detokenize, and needle_get_version—defined in needle/model/export.py to enable zero-overhead inference from C, C++, Rust, and other languages without requiring a Python runtime.

The cactus-compute/needle repository provides a high-performance inference engine built on GGML/GGUF quantization. For system-level integrations, the Needle 2 FFI functions wrap the engine's Python internals via CFFI bindings, allowing external programs to load quantized checkpoints, manage tokenization, and execute generation loops through a stable C Application Binary Interface (ABI).

Core FFI Function Reference

All foreign function interface definitions reside in needle/model/export.py, which exports a C-compatible shared library (libneedle_ffi.so on Linux, needle_ffi.dll on Windows, or libneedle_ffi.dylib on macOS). The following table documents the complete public API surface:

Function	Purpose	C Signature
needle_load_model	Loads a quantized GGML/GGUF checkpoint and tokenizer into memory, returning an opaque handle.	void* needle_load_model(const char* checkpoint_path, const char* tokenizer_path, const char* device)
needle_unload_model	Releases all resources associated with a model handle.	void needle_unload_model(void* model_handle)
needle_set_params	Configures generation parameters (temperature, top-p, max tokens) for a specific model instance.	int needle_set_params(void* model_handle, const needle_params_t* params)
needle_infer	Executes a single forward pass on tokenized input and returns the next token ID.	int needle_infer(void* model_handle, const int32_t* input_ids, size_t input_len, int32_t* output_id)
needle_generate	Performs autoregressive generation for a complete sequence, writing token IDs to an output buffer.	int needle_generate(void* model_handle, const int32_t* prompt_ids, size_t prompt_len, int32_t* out_ids, size_t* out_len)
needle_tokenize	Converts UTF-8 text into integer token IDs using the model's bundled SentencePiece or BPE tokenizer.	int needle_tokenize(const char* text, int32_t* token_buf, size_t buf_len, size_t* token_len)
needle_detokenize	Converts an array of token IDs back into a null-terminated UTF-8 string.	int needle_detokenize(const int32_t* tokens, size_t token_len, char* out_str, size_t out_buf_len)
needle_get_version	Returns the library version string (e.g., "2.0.11").	const char* needle_get_version(void)

All functions return integer status codes where 0 indicates success and non-zero values signal errors (invalid handle, buffer overflow, or generation failure).

Implementation Architecture

The FFI layer in needle/model/export.py serves as a thin translation shim between C types and the high-level Python inference logic. When you invoke needle_load_model, the wrapper:

1. Validates input paths and device strings ("cpu", "cuda", "metal").
2. Instantiates the Model class defined in needle/model/run.py, which handles GGML tensor allocation and KV-cache management.
3. Loads the tokenizer implementation from needle/model/tokenizer.py.
4. Returns an opaque void* pointer to the Python object, cast for C compatibility.

The generation functions (needle_generate and needle_infer) delegate to the sampling and forward-pass methods in needle/model/run.py, while tokenization calls utilize the encoding/decoding logic maintained in needle/model/tokenizer.py. This architecture ensures that the C interface automatically benefits from optimizations in the underlying GGML compute graph and quantization kernels defined in needle/model/architecture.py.

Integration Examples

C Application Integration

The following complete example demonstrates loading a quantized model, tokenizing a prompt, generating a response, and detokenizing the output. The code assumes you have generated the needle_ffi.h header during the Python package installation (pip install needle).

#include <stdio.h>
#include <stdint.h>
#include "needle_ffi.h"

int main(void) {
    // Initialize model with CPU backend (use "cuda" or "metal" for GPU acceleration)
    void *model = needle_load_model(
        "models/ggml-model-q8_0.bin",
        "tokenizer.model",
        "cpu"
    );
    if (!model) {
        fprintf(stderr, "Failed to load model checkpoint\n");
        return 1;
    }

    // Tokenize input prompt
    const char *prompt = "Explain quantum computing in one sentence.";
    int32_t prompt_ids[256];
    size_t prompt_len = 0;
    
    if (needle_tokenize(prompt, prompt_ids, 256, &prompt_len) != 0) {
        fprintf(stderr, "Tokenization failed\n");
        needle_unload_model(model);
        return 1;
    }

    // Generate up to 64 new tokens
    int32_t output_ids[320];
    size_t output_len = 0;
    
    if (needle_generate(model, prompt_ids, prompt_len, 
                       output_ids, &output_len) != 0) {
        fprintf(stderr, "Generation failed\n");
        needle_unload_model(model);
        return 1;
    }

    // Convert token IDs back to text
    char result[1024];
    if (needle_detokenize(output_ids, output_len, 
                         result, sizeof(result)) != 0) {
        fprintf(stderr, "Detokenization failed\n");
        needle_unload_model(model);
        return 1;
    }

    printf("Generated text: %s\n", result);
    
    // Cleanup
    needle_unload_model(model);
    return 0;
}

Key implementation notes:

• Memory ownership: The caller manages all buffer allocations; needle_load_model returns the only heap-allocated object that the caller does not own, which must be freed via needle_unload_model.
• Device selection: Pass "cpu" for GGML CPU inference, "cuda" for NVIDIA GPUs, or "metal" for Apple Silicon.
• Error handling: Always check return codes; needle_generate returns non-zero if the requested out_len exceeds available buffer space.

Rust Dynamic Loading

For Rust applications requiring runtime linking (avoiding static compilation against the FFI), use the libloading crate to resolve symbols from the shared library. This pattern matches how the Python package distributes its compiled artifacts.

use libloading::{Library, Symbol};
use std::ffi::{CStr, CString};
use std::os::raw::{c_char, c_int, c_void};

type LoadModel = unsafe extern "C" fn(*const c_char, *const c_char, *const c_char) -> *mut c_void;
type UnloadModel = unsafe extern "C" fn(*mut c_void);
type Tokenize = unsafe extern "C" fn(*const c_char, *mut i32, usize, *mut usize) -> c_int;
type Generate = unsafe extern "C" fn(*mut c_void, *const i32, usize, *mut i32, *mut usize) -> c_int;
type Detokenize = unsafe extern "C" fn(*const i32, usize, *mut c_char, usize) -> c_int;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let lib = Library::new("libneedle_ffi.so")?;

    unsafe {
        let load: Symbol<LoadModel> = lib.get(b"needle_load_model")?;
        let unload: Symbol<UnloadModel> = lib.get(b"needle_unload_model")?;
        let tokenize: Symbol<Tokenize> = lib.get(b"needle_tokenize")?;
        let generate: Symbol<Generate> = lib.get(b"needle_generate")?;
        let detokenize: Symbol<Detokenize> = lib.get(b"needle_detokenize")?;

        // Load model
        let model = load(
            CString::new("models/ggml-model-q4_0.bin")?.as_ptr(),
            CString::new("tokenizer.json")?.as_ptr(),
            CString::new("cpu")?.as_ptr(),
        );
        assert!(!model.is_null());

        // Tokenize
        let prompt = CString::new("Write a haiku about rust.")?;
        let mut ids = [0i32; 128];
        let mut len: usize = 0;
        assert_eq!(0, tokenize(prompt.as_ptr(), ids.as_mut_ptr(), ids.len(), &mut len as *mut usize));

        // Generate
        let mut out = [0i32; 256];
        let mut out_len: usize = 0;
        assert_eq!(0, generate(model, ids.as_ptr(), len, out.as_mut_ptr(), &mut out_len as *mut usize));

        // Detokenize
        let mut buf = [0u8; 1024];
        assert_eq!(0, detokenize(out.as_ptr(), out_len, buf.as_mut_ptr() as *mut c_char, buf.len()));
        
        println!("Output: {}", CStr::from_ptr(buf.as_ptr() as *const c_char).to_string_lossy());
        
        unload(model);
    }

    Ok(())
}

Key implementation notes:

• Type safety: Define unsafe extern "C" function pointers matching the signatures exported from export.py.
• String handling: Convert Rust String types to CString to ensure null-termination for the C ABI boundary.
• Resource cleanup: Call needle_unload_model before dropping the Library handle to prevent dangling pointers.

Summary

• Eight core functions comprise the complete Needle 2 FFI surface: model lifecycle management (needle_load_model, needle_unload_model), generation control (needle_set_params, needle_infer, needle_generate), text processing (needle_tokenize, needle_detokenize), and metadata (needle_get_version).
• Source location: All bindings are implemented in needle/model/export.py, which wraps the Python inference logic found in needle/model/run.py and tokenization utilities in needle/model/tokenizer.py.
• Library naming: Link against libneedle_ffi.so (Linux), needle_ffi.dll (Windows), or libneedle_ffi.dylib (macOS) generated during the Python package build.
• Memory model: The FFI uses a handle-based design where void* model_handle represents the Python Model instance; all other buffers are caller-allocated.
• Format support: The interface specifically handles GGML and GGUF quantized checkpoints, with automatic device selection for CPU, CUDA, and Metal backends.

Frequently Asked Questions

Which shared library file do I need to link against?

Link against libneedle_ffi.so on Linux systems, needle_ffi.dll on Windows, or libneedle_ffi.dylib on macOS. These files are compiled automatically when you install the needle package via pip, typically located in the package's installation directory under needle/lib/.

Do I need to install Python to use the Needle 2 FFI functions at runtime?

No. While the shared library is generated from Python source code using CFFI, the resulting binary is a standalone shared object with no runtime dependency on libpython or the Python interpreter. You only need Python during the build/installation phase to generate the library file.

What model formats are supported by needle_load_model?

The function accepts GGML and GGUF quantized checkpoint files produced by the GGML conversion tools. You must also provide the corresponding tokenizer file (SentencePiece .model or HuggingFace tokenizer.json) as the second argument. The implementation in needle/model/export.py validates the file headers before passing them to the GGML loader in needle/model/run.py.

How does memory management work for the opaque model_handle?

The void* pointer returned by needle_load_model references a Python Model object retained in memory by the CFFI layer. You must explicitly call needle_unload_model to decrement the reference count and free associated GGML tensors, KV caches, and tokenizer objects. Failure to call this function results in memory leaks. All other buffers (token arrays, output strings) are allocated and freed by the caller.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}