How to Integrate Custom AI Inference Models into ApraPipes Transforms: A Complete Guide

To integrate a custom AI inference model into ApraPipes, create a Props class for configuration, implement a Transform module that loads the model in init() and runs inference in process(), then register it in ModuleRegistrations.cpp using filePathProp for the model path.

ApraPipes is a high-performance C++ framework for building media processing pipelines. Integrating custom AI inference models into ApraPipes transforms allows you to embed ONNX Runtime, TensorRT, or PyTorch models directly into your declarative pipelines. This guide walks through the exact implementation pattern used by built-in modules like AudioToTextXForm and FaceDetectorXform in the apra-labs/aprapipes repository.

Understanding the ApraPipes Module Architecture

ApraPipes treats every processing step as a Module. A Transform module consumes frames, runs user-defined logic such as AI inference, and emits new frames. The architecture separates configuration from implementation through Props classes that describe configurable parameters like modelPath and batchSize.

Built-in transforms demonstrate this pattern clearly. AudioToTextXForm loads OpenAI's Whisper model in init(), runs speech recognition in process(), and emits text frames. FaceDetectorXform uses OpenCV DNN to load Caffe models for face detection. Your custom implementation will follow identical lifecycle methods.

Step-by-Step Implementation Guide

Step 1 – Define the Props Class

Create a Props class that inherits from ModuleProps to expose configuration options. Use the property-binding helpers so the declarative pipeline can populate them from JSON.

In base/include/MyModelXForm.h, define properties with applyProperties and getProperty methods:

class MyModelXFormProps : public ModuleProps {
public:
    std::string modelPath;
    int batchSize = 1;

    template <typename PropsT>
    static void applyProperties(PropsT& props,
        const std::map<std::string, apra::ScalarPropertyValue>& values,
        std::vector<std::string>& missingRequired) {
        apra::applyProp(props.modelPath, "modelPath", values, true, missingRequired);
        apra::applyProp(props.batchSize, "batchSize", values, false, missingRequired);
    }

    apra::ScalarPropertyValue getProperty(const std::string& name) const {
        if (name == "modelPath") return modelPath;
        if (name == "batchSize") return static_cast<int64_t>(batchSize);
        throw std::runtime_error("Unknown property: " + name);
    }
};

Reference AudioToTextXForm.h in the repository to see the complete AudioToTextXFormProps definition and the applyProperties template pattern.

Step 2 – Implement the Transform Class

Implement the Transform class in base/src/MyModelXForm.cpp by deriving from Module. Store a Detail object that holds the model handle, and implement the lifecycle methods:

• init() – Load the model using the path from Props (e.g., ONNX Runtime, TensorRT).
• process() – Convert incoming frames to input tensors, run inference, decode results, and push output frames.
• validateInputPins() / validateOutputPins() – Declare expected frame types (e.g., RawImage, Tensor, Text).

bool MyModelXForm::init() {
    Ort::SessionOptions opts;
    opts.SetIntraOpNumThreads(1);
    mDetail->session = new Ort::Session(mDetail->ortEnv,
        mDetail->props.modelPath.c_str(), opts);
    return Module::init();
}

bool MyModelXForm::process(frame_container& frames) {
    auto inFrame = frames.begin()->second;
    
    // Convert frame to tensor and run inference
    std::array<const char*, 1> inputNames = {"input"};
    std::array<const char*, 1> outputNames = {"output"};
    auto outputTensors = mDetail->session->Run(Ort::RunOptions{}, 
        inputNames.data(), &inputTensor, 1, outputNames.data(), 1);
    
    // Decode and emit result
    std::string result = decodeOutput(outputTensors.front());
    auto outFrame = makeFrame(result.size());
    memcpy(outFrame->data(), result.c_str(), result.size());
    frames.insert({mDetail->outputPinId, outFrame});
    send(frames);
    return true;
}

Study AudioToTextXForm.cpp to see how whisper_init_from_file_with_params loads the Whisper model and how the process() method handles inference.

Step 3 – Register the Module

Register the new module in base/src/declarative/ModuleRegistrations.cpp. Add a block similar to AudioToTextXForm, specifying category, description, input/output pins, and property definitions. Use filePathProp with PathRequirement::MustExist to ensure the model file is present at pipeline load time.

if (!registry.hasModule("MyModelXForm")) {
    registerModule<MyModelXForm, MyModelXFormProps>()
        .category(ModuleCategory::Transform)
        .description("Runs custom AI inference (ONNX/TensorRT) on input frames")
        .tags("transform", "ai", "inference", "onnx", "ml")
        .input("input", "RawImage")
        .output("output", "Text")
        .filePathProp("modelPath", "Path to the model file (ONNX / TensorRT)",
                      PathRequirement::MustExist, true)
        .intProp("batchSize", "Batch size for inference", false, 1, 1, 128)
        .selfManagedOutputPins();
}

Lines 1042-1054 in ModuleRegistrations.cpp demonstrate the registration pattern for AudioToTextXForm with filePathProp.

Step 4 – Add Documentation and Tests

Create documentation in docs/declarative-pipeline/MyModelXForm.md describing the required properties and example pipeline JSON. Update docs/declarative-pipeline/PROGRESS.md to list the new module so the web UI picks it up.

Add a unit test in base/test/ mirroring existing XForm tests like audioToTextXform_tests.cpp. The test should load a tiny test model and verify output frame generation. Re-run the build with cmake && cmake --build build to verify compilation across all CI workflows (Linux, Windows, macOS, ARM64).

How the Components Work Together

When a JSON pipeline is parsed, the ModuleFactory looks up the module name (MyModelXForm) in the ModuleRegistry populated by ModuleRegistrations.cpp. The JSON object's modelPath field maps to MyModelXFormProps::modelPath via the applyProperties template, ensuring type-safe configuration binding.

At runtime, init() executes once per module instance, loading the model from the supplied path—mirroring how AudioToTextXForm loads Whisper via whisper_init_from_file_with_params. For each incoming frame, process() converts the frame to the model's input tensor, executes inference, decodes the results, and forwards the output frame downstream using send(frames).

Key Files to Reference

• base/include/AudioToTextXForm.h – Props definition and applyProperties template pattern
• base/src/AudioToTextXForm.cpp – Model loading (whisper_init_from_file_with_params) and inference flow in process()
• base/src/declarative/ModuleRegistrations.cpp – Module registration with filePathProp and PathRequirement::MustExist (lines 1042-1054)
• base/src/FaceDetectorXform.cpp – Deep learning model loading (Caffe) using OpenCV DNN
• docs/declarative-pipeline/PROGRESS.md – Master list of registered modules for UI integration

Summary

• Create a Props class inheriting from ModuleProps with modelPath and configuration fields, implementing applyProperties for JSON binding.
• Implement the Transform by deriving from Module, loading the model in init(), and running inference in process() using your chosen SDK (ONNX Runtime, TensorRT, etc.).
• Register the module in ModuleRegistrations.cpp using filePathProp with PathRequirement::MustExist to ensure the model file is available at pipeline load time.
• Document and test by adding markdown documentation in docs/declarative-pipeline/ and unit tests in base/test/ to verify functionality across all supported platforms.

Frequently Asked Questions

What inference engines are supported for custom AI models in ApraPipes?

ApraPipes supports any inference engine that provides a C++ API. The framework itself is agnostic to the backend implementation. You can integrate ONNX Runtime, TensorRT, OpenCV DNN, PyTorch C++ (LibTorch), or proprietary SDKs by including their headers in your Transform implementation and linking the appropriate libraries in your CMake configuration.

How do I handle different input frame types in my custom transform?

Override the validateInputPins() method to declare acceptable frame types. For example, return meta->getFrameType() == FrameMetadata::RAW_IMAGE for image inputs or FrameMetadata::AUDIO for audio. In process(), cast the incoming frame to the appropriate type using getFrame() or similar accessors. If your model requires tensor inputs, convert the frame data using utility functions from FrameMetadata or Utils.h before feeding it to the inference session.

Can I use GPU acceleration for custom AI inference in ApraPipes?

Yes. When registering your module in ModuleRegistrations.cpp, use the registerCudaModule helper instead of registerModule if your implementation requires CUDA streams. In your Detail class or init() method, configure your inference engine for GPU execution—for example, create an ONNX Runtime session with OrtCUDAProviderOptions, or initialize TensorRT with CUDA context. Ensure your process() method handles GPU memory appropriately and synchronizes CUDA streams before sending frames downstream.

How do I debug inference errors in my custom transform?

Add detailed logging in the init() method to verify model loading success and path resolution. In process(), check the return values of inference API calls (e.g., Ort::Session::Run) and throw exceptions or return false to signal pipeline failure. Use the validateInputPins() and validateOutputPins() methods to catch metadata mismatches early. For runtime debugging, build with cmake -DCMAKE_BUILD_TYPE=Debug and use GDB or Visual Studio to step through the process() method, inspecting tensor shapes and frame data before and after inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}