GPU Memory Management in ApraPipes: Best Practices for High-Performance Pipelines

Use ApraPipes' per-PIN memory-type model to keep data on the GPU automatically, allocate pinned host memory for uploads, and always retain the primary CUDA context instead of creating per-instance contexts to prevent memory leaks.

ApraPipes implements a declarative pipeline framework where every module's input and output pins explicitly declare their memory location. This design enables automatic bridge insertion and optimal GPU memory management without manual memory copies. Understanding these patterns is essential for building high-throughput video processing pipelines that avoid costly PCIe transfers and GPU memory leaks.

Understanding the Per-PIN Memory Type Model

ApraPipes uses a strict per-PIN memory-type model where each pin declares one of four locations: HOST, HOST_PINNED, CUDA_DEVICE, or DMABUF. This explicit declaration allows the framework to validate memory compatibility before init() runs and automatically insert transfer bridges only when necessary.

In base/include/declarative/Metadata.h, the PinDef struct captures this through the memType field:

struct PinDef {
    std::string name;
    FrameType frameType;
    FrameMetadata::MemType memType;  // HOST, HOST_PINNED, CUDA_DEVICE, DMABUF
    std::vector<ImageType> imageTypes;
};

When building pipelines declaratively, the PipelineAnalyzer inspects these memType values to determine if a CudaMemCopy bridge is required between modules. This validation happens at configuration time, preventing runtime memory mismatches.

Automatic Bridge Insertion and Zero-Copy Optimization

The framework automatically inserts memory transfer modules to maintain data locality on the GPU. According to the design documentation in docs/declarative-pipeline/CUDA_MEMTYPE_DESIGN.md, the PipelineAnalyzer inserts CudaMemCopy bridges for HOST↔CUDA_DEVICE transfers and ColorConversion or CCNPPI modules for format conversions when needed.

This architecture allows pipelines to stay on the GPU across multiple processing stages without explicit user intervention. For example, when connecting a HOST_PINNED source to a GaussianBlur module (which requires CUDA_DEVICE), the analyzer automatically inserts a CudaMemCopy bridge with kind: HostToDevice.

To avoid redundant transfers, the analyzer emits warnings when detecting patterns like Host→CudaCopy→Host→CudaCopy. Refactor such pipelines to use GPU-accelerated alternatives (e.g., replacing CPU filters with CUDA counterparts) to eliminate intermediate device-to-host copies.

Pinned Host Memory for Efficient Data Transfers

For data that must originate from host memory, always use page-locked (pinned) allocations instead of standard pageable memory. Pinned memory enables asynchronous DMA transfers and achieves the full PCIe bandwidth for host-to-device copies.

Using apra_cudamallochost_allocator

The apra_cudamallochost_allocator class in base/src/apra_cudamallochost_allocator.cu provides a pool-based allocator for pinned host buffers:

#include "apra_cudamallochost_allocator.h"

// Allocate pinned memory for a 1080p BGR frame
const size_t frameBytes = 1920 * 1080 * 3;
apra_cudamallochost_allocator allocator;
char* pinnedBuf = static_cast<char*>(allocator.malloc(frameBytes));

// Use in pipeline - the framework auto-inserts CudaMemCopy bridge
// ... processing ...
allocator.free(pinnedBuf);

Using HOST_PINNED rather than HOST reduces PCI-e latency and improves throughput for large video frames, particularly when feeding high-resolution content into GPU encoders like JPEGEncoderNVJPEG.

CUDA Context Management Best Practices

Proper CUDA context lifecycle management is critical for preventing GPU memory leaks in long-running applications and unit tests. ApraPipes enforces specific patterns for context creation and destruction.

Primary Context vs. Per-Instance Contexts

Never call cuCtxCreate directly in module code. Instead, retain the primary context using cuDevicePrimaryCtxRetain, which allows the CUDA driver to manage context sharing across modules. This pattern, implemented in base/src/H264DecoderNvCodecHelper.cpp, prevents per-instance GPU memory leaks that previously caused OOM failures in CI test suites.

// Correct pattern from H264DecoderNvCodecHelper.cpp
CUcontext ctx = nullptr;
CUDA_DRVAPI_CALL(cuDevicePrimaryCtxRetain(&ctx, m_cuDevice));

// Initialize decoder with ctx...
NvDecoder decoder(loader, ctx, cudaVideoCodec_H264, ...);

// Release on destruction
CUDA_DRVAPI_CALL(cuDevicePrimaryCtxRelease(m_cuDevice));

The primary context remains active until all retentions are released, eliminating the overhead of repeated context creation and destruction.

Explicit Resource Destruction

Always explicitly destroy driver resources in module destructors. The CuCtxSynchronize class in base/src/CuCtxSynchronize.cpp demonstrates proper synchronization and cleanup patterns. When modules are created repeatedly (e.g., in unit tests), failing to release contexts causes hidden GPU memory accumulation.

Use the CUDA_DRVAPI_CALL macro (defined in the CUDA driver loader) to ensure all driver API calls are checked and abort on failure, preventing undefined behavior from unchecked error codes.

Shared CUDA Streams and Synchronization

ApraPipes uses a single shared CUDA stream created by ModuleFactory in base/src/declarative/ModuleFactory.cpp. All CUDA modules and auto-inserted bridges receive this cudastream_sp, removing per-module stream creation overhead and guaranteeing ordered execution of GPU work.

Do not construct separate cudaStream_t instances unless absolutely necessary for specialized asynchronous operations. The shared stream ensures proper synchronization across the entire pipeline while minimizing resource consumption.

Registering CUDA Modules with Correct Memory Types

When adding new GPU-accelerated modules, explicitly declare CUDA_DEVICE memory types in base/src/declarative/ModuleRegistrations.cpp. This registration serves as the single source of truth for the analyzer:

// In ModuleRegistrations.cpp
ModuleRegistrationBuilder<MyCustomCuda>("MyCustomCuda")
    .inputPin("input", FrameType::RAW_IMAGE, FrameMetadata::CUDA_DEVICE)
    .outputPin("output", FrameType::RAW_IMAGE, FrameMetadata::CUDA_DEVICE)
    .tag("function:custom")
    .tag("backend:cuda");

Once registered, any pipeline connecting a CPU-only source to MyCustomCuda automatically receives the necessary CudaMemCopy bridge. For manual transfers in custom preprocessing steps, explicitly insert a CudaMemCopy module in the JSON configuration:

{
    "modules": {
        "upload": { "type": "CudaMemCopy", "props": { "kind": "HostToDevice" } },
        "process": { "type": "MyCustomCuda" }
    },
    "connections": [
        { "from": "source", "to": "upload" },
        { "from": "upload", "to": "process" }
    ]
}

Detecting and Preventing Memory Leaks

The test harness tracks GPU allocations via the driver's reference counting. Run with --detect_memory_leaks=0 only for debugging specific allocation patterns. For production code, ensure every cuDevicePrimaryCtxRetain has a matching cuDevicePrimaryCtxRelease in the destructor.

Query the module registry to find GPU-accelerated alternatives and avoid suboptimal CPU-only modules:

auto gpuResizes = registry.getModulesWithTags(
    {"function:resize", "backend:cuda"}); // Returns ["ResizeNPPI"]
if (!gpuResizes.empty()) {
    // Use GPU variant to avoid Host→Device transfers
}

Summary

• Declare explicit memType on every PIN in ModuleRegistrations.cpp to enable automatic validation and bridge insertion.
• Use HOST_PINNED allocations via apra_cudamallochost_allocator for all host-side buffers that feed GPU modules.
• Retain the primary CUDA context using cuDevicePrimaryCtxRetain rather than cuCtxCreate to prevent per-instance memory leaks.
• Leverage the shared CUDA stream from ModuleFactory instead of creating custom streams for standard processing.
• Allow the PipelineAnalyzer to insert bridges automatically rather than manually managing CudaMemCopy modules unless specifically required.
• Explicitly release all CUDA contexts in destructors to prevent accumulation during repeated module instantiation in tests.

Frequently Asked Questions

How does ApraPipes automatically manage GPU memory transfers?

ApraPipes uses a per-PIN memory-type model where each module declares its input and output memory locations (HOST, HOST_PINNED, CUDA_DEVICE, or DMABUF) via PinDef::memType in Metadata.h. The PipelineAnalyzer inspects these declarations and automatically inserts CudaMemCopy bridges when connecting modules with mismatched memory types, ensuring data reaches the GPU without manual intervention while keeping frames on the device across compatible modules.

What is the correct way to allocate host memory for GPU uploads in ApraPipes?

Use the apra_cudamallochost_allocator class defined in base/src/apra_cudamallochost_allocator.cu to allocate page-locked (pinned) host memory. Pinned memory enables asynchronous DMA transfers and maximizes PCIe bandwidth for host-to-device copies. Allocate with allocator.malloc(size), use the buffer to fill frame data, and the framework will automatically bridge the pinned memory to CUDA modules.

Why should I use cuDevicePrimaryCtxRetain instead of cuCtxCreate?

Using cuDevicePrimaryCtxRetain (as implemented in base/src/H264DecoderNvCodecHelper.cpp) prevents per-instance GPU memory leaks that occur when modules repeatedly create and destroy contexts. The primary context is shared across the application and managed by the CUDA driver, eliminating OOM failures in long-running test suites and production pipelines that dynamically create decoder or encoder instances.

How do I register a new CUDA module to work with the automatic bridge system?

Register the module in base/src/declarative/ModuleRegistrations.cpp using ModuleRegistrationBuilder, explicitly setting FrameMetadata::CUDA_DEVICE for both input and output pins. This declaration allows the PipelineAnalyzer to detect when your module requires GPU memory and automatically insert the necessary CudaMemCopy bridges from upstream HOST or HOST_PINNED sources.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}