Core C FFI Functions in Needle: The Complete Python-to-C Interface

Needle exposes exactly four low-level C functions—engine_load, engine_forward, tokenizer_encode, and tokenizer_decode—through a lightweight CFFI layer in needle/model/run.py to enable high-performance on-device inference.

Needle is a minimal inference engine from Cactus Compute that delivers a 28 MiB runtime for running compiled .cact models. The repository implements a thin Python wrapper that binds to optimized C code using CFFI, exposing only the essential C FFI functions required for model execution and tokenization.

The Four Core C FFI Functions Bound in needle/model/run.py

The entire Python-to-C interface lives in needle/model/run.py and exposes four critical operations. The wrapper initializes these bindings at import time using cffi.FFI() and maps them to user-friendly Python functions.

engine_load – Initializing the Inference Context

The first C FFI function, engine_load, handles model initialization. According to the source code in needle/model/run.py, this function accepts a file path to a .cact checkpoint and returns a pointer to the engine context.

engine_forward – Executing the Inference Step

The engine_forward function performs the actual computation. It accepts token IDs, the KV cache pointer, and output buffers for logits. This C FFI function updates the cache in-place and returns prediction scores without Python overhead during the forward pass.

tokenizer_encode and tokenizer_decode – Text Processing

Text conversion happens through two dedicated C FFI functions. tokenizer_encode converts UTF-8 strings to integer token arrays, while tokenizer_decode reverses the process. Both operate directly on C strings to avoid Python string copying overhead.

How the FFI Layer Initializes the Shared Library

The binding mechanism uses CFFI's ABI mode. When needle/model/run.py loads, it executes:

_ffi = cffi.FFI()
_ffi.cdef("""
    void* engine_load(const char* path);
    void  engine_forward(void* ctx,
                         const int32_t* tokens, size_t n_tokens,
                         float* logits, size_t vocab_size,
                         void* kv_cache);
    int32_t* tokenizer_encode(const char* text, size_t* out_len);
    char*   tokenizer_decode(const int32_t* tokens, size_t n);
""")
_lib = _ffi.dlopen(_LIB_PATH)

This declares the C function signatures once, then dynamically loads the compiled shared library at _LIB_PATH. The Python wrapper functions convert Python objects to raw C pointers before delegating to these bound symbols.

Complete Workflow Example

The following code demonstrates the four core C FFI functions in action:

import needle

# 1️⃣ Load a tuned model (the .cact file contains the compiled engine)

engine = needle.load_engine("my_needle.cact")

# 2️⃣ Encode a prompt – uses the C tokenizer

tokens = needle.encode("Translate English to French: Hello world.")

# 3️⃣ Run a forward pass – calls the C inference routine

logits, new_state = needle.run_engine(engine, tokens)

# 4️⃣ Decode the generated tokens back to text

response = needle.decode(logits.argmax(axis=-1))
print(response)     # → "Bonjour le monde."

Each public function—load_engine, run_engine, encode, and decode—forwards to its corresponding C implementation with minimal marshalling overhead.

Architecture of the C FFI Surface

The C FFI functions are supported by four key files that constitute the minimal interface:

• needle/model/run.py: Contains the CFFI declarations and the Python wrappers that bind the core engine functions.
• needle/model/export.py: Exports PyTorch checkpoints to the single-file .cact binary that engine_load reads.
• needle/model/architecture.py: Defines the neural network layout that the compiled C engine implements.
• needle/__init__.py: Exposes public entry points that forward to the FFI layer in run.py.

Summary

• Needle exposes exactly four C FFI functions: engine_load, engine_forward, tokenizer_encode, and tokenizer_decode.
• The bindings are defined in needle/model/run.py using CFFI and loaded via _ffi.dlopen().
• Each function handles a critical path: model loading, inference execution, and text tokenization.
• The deliberately tiny C API keeps the runtime footprint at 28 MiB while eliminating Python overhead during the forward pass.

Frequently Asked Questions

What are the exact C function signatures exposed by Needle's FFI?

Needle declares four functions in its CFFI header: void* engine_load(const char* path), void engine_forward(void* ctx, const int32_t* tokens, size_t n_tokens, float* logits, size_t vocab_size, void* kv_cache), int32_t* tokenizer_encode(const char* text, size_t* out_len), and char* tokenizer_decode(const int32_t* tokens, size_t n). These signatures are declared once in needle/model/run.py using _ffi.cdef().

How does Needle minimize overhead when calling C FFI functions from Python?

The wrapper in needle/model/run.py uses CFFI's ABI mode to declare signatures once at import time, then calls _ffi.dlopen() to load the shared library. Python wrapper functions convert objects to raw pointers before crossing the language boundary, eliminating data copying during the inference hot path.

Where is the C library actually loaded in the Needle codebase?

The shared library is loaded in needle/model/run.py through the line _lib = _ffi.dlopen(_LIB_PATH), which executes when the module is first imported. This binds the Python names to the actual C symbols for the duration of the process.

Why does Needle use CFFI instead of ctypes or Cython for its Python bindings?

CFFI provides a minimal, explicit declaration of C function signatures that matches Needle's design philosophy of a tiny runtime footprint. The _ffi.cdef() approach generates efficient calling code without requiring Cython compilation steps or ctypes' runtime signature lookup overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}