# GPU Memory Management in ApraPipes: Best Practices for High-Performance Pipelines

> Master GPU memory management in ApraPipes. Learn best practices for high-performance pipelines including pinned host memory usage and context management to prevent leaks.

- Repository: [Apra Labs/aprapipes](https://github.com/apra-labs/aprapipes)
- Tags: best-practices
- Published: 2026-02-25

---

**Use ApraPipes' per-PIN memory-type model to keep data on the GPU automatically, allocate pinned host memory for uploads, and always retain the primary CUDA context instead of creating per-instance contexts to prevent memory leaks.**

ApraPipes implements a declarative pipeline framework where every module's input and output pins explicitly declare their memory location. This design enables automatic bridge insertion and optimal GPU memory management without manual memory copies. Understanding these patterns is essential for building high-throughput video processing pipelines that avoid costly PCIe transfers and GPU memory leaks.

## Understanding the Per-PIN Memory Type Model

ApraPipes uses a strict **per-PIN memory-type model** where each pin declares one of four locations: `HOST`, `HOST_PINNED`, `CUDA_DEVICE`, or `DMABUF`. This explicit declaration allows the framework to validate memory compatibility before `init()` runs and automatically insert transfer bridges only when necessary.

In [`base/include/declarative/Metadata.h`](https://github.com/apra-labs/aprapipes/blob/main/base/include/declarative/Metadata.h), the `PinDef` struct captures this through the `memType` field:

```cpp
struct PinDef {
    std::string name;
    FrameType frameType;
    FrameMetadata::MemType memType;  // HOST, HOST_PINNED, CUDA_DEVICE, DMABUF
    std::vector<ImageType> imageTypes;
};

```

When building pipelines declaratively, the `PipelineAnalyzer` inspects these `memType` values to determine if a `CudaMemCopy` bridge is required between modules. This validation happens at configuration time, preventing runtime memory mismatches.

## Automatic Bridge Insertion and Zero-Copy Optimization

The framework automatically inserts memory transfer modules to maintain data locality on the GPU. According to the design documentation in [`docs/declarative-pipeline/CUDA_MEMTYPE_DESIGN.md`](https://github.com/apra-labs/aprapipes/blob/main/docs/declarative-pipeline/CUDA_MEMTYPE_DESIGN.md), the `PipelineAnalyzer` inserts `CudaMemCopy` bridges for HOST↔CUDA_DEVICE transfers and `ColorConversion` or `CCNPPI` modules for format conversions when needed.

This architecture allows pipelines to stay on the GPU across multiple processing stages without explicit user intervention. For example, when connecting a `HOST_PINNED` source to a `GaussianBlur` module (which requires `CUDA_DEVICE`), the analyzer automatically inserts a `CudaMemCopy` bridge with `kind: HostToDevice`.

To avoid redundant transfers, the analyzer emits warnings when detecting patterns like `Host→CudaCopy→Host→CudaCopy`. Refactor such pipelines to use GPU-accelerated alternatives (e.g., replacing CPU filters with CUDA counterparts) to eliminate intermediate device-to-host copies.

## Pinned Host Memory for Efficient Data Transfers

For data that must originate from host memory, **always use page-locked (pinned) allocations** instead of standard pageable memory. Pinned memory enables asynchronous DMA transfers and achieves the full PCIe bandwidth for host-to-device copies.

### Using apra_cudamallochost_allocator

The `apra_cudamallochost_allocator` class in `base/src/apra_cudamallochost_allocator.cu` provides a pool-based allocator for pinned host buffers:

```cpp
#include "apra_cudamallochost_allocator.h"

// Allocate pinned memory for a 1080p BGR frame
const size_t frameBytes = 1920 * 1080 * 3;
apra_cudamallochost_allocator allocator;
char* pinnedBuf = static_cast<char*>(allocator.malloc(frameBytes));

// Use in pipeline - the framework auto-inserts CudaMemCopy bridge
// ... processing ...
allocator.free(pinnedBuf);

```

Using `HOST_PINNED` rather than `HOST` reduces PCI-e latency and improves throughput for large video frames, particularly when feeding high-resolution content into GPU encoders like `JPEGEncoderNVJPEG`.

## CUDA Context Management Best Practices

Proper CUDA context lifecycle management is critical for preventing GPU memory leaks in long-running applications and unit tests. ApraPipes enforces specific patterns for context creation and destruction.

### Primary Context vs. Per-Instance Contexts

**Never call `cuCtxCreate` directly in module code.** Instead, retain the primary context using `cuDevicePrimaryCtxRetain`, which allows the CUDA driver to manage context sharing across modules. This pattern, implemented in [`base/src/H264DecoderNvCodecHelper.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/H264DecoderNvCodecHelper.cpp), prevents per-instance GPU memory leaks that previously caused OOM failures in CI test suites.

```cpp
// Correct pattern from H264DecoderNvCodecHelper.cpp
CUcontext ctx = nullptr;
CUDA_DRVAPI_CALL(cuDevicePrimaryCtxRetain(&ctx, m_cuDevice));

// Initialize decoder with ctx...
NvDecoder decoder(loader, ctx, cudaVideoCodec_H264, ...);

// Release on destruction
CUDA_DRVAPI_CALL(cuDevicePrimaryCtxRelease(m_cuDevice));

```

The primary context remains active until all retentions are released, eliminating the overhead of repeated context creation and destruction.

### Explicit Resource Destruction

Always explicitly destroy driver resources in module destructors. The `CuCtxSynchronize` class in [`base/src/CuCtxSynchronize.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/CuCtxSynchronize.cpp) demonstrates proper synchronization and cleanup patterns. When modules are created repeatedly (e.g., in unit tests), failing to release contexts causes hidden GPU memory accumulation.

Use the `CUDA_DRVAPI_CALL` macro (defined in the CUDA driver loader) to ensure all driver API calls are checked and abort on failure, preventing undefined behavior from unchecked error codes.

## Shared CUDA Streams and Synchronization

ApraPipes uses a **single shared CUDA stream** created by `ModuleFactory` in [`base/src/declarative/ModuleFactory.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/declarative/ModuleFactory.cpp). All CUDA modules and auto-inserted bridges receive this `cudastream_sp`, removing per-module stream creation overhead and guaranteeing ordered execution of GPU work.

Do not construct separate `cudaStream_t` instances unless absolutely necessary for specialized asynchronous operations. The shared stream ensures proper synchronization across the entire pipeline while minimizing resource consumption.

## Registering CUDA Modules with Correct Memory Types

When adding new GPU-accelerated modules, explicitly declare `CUDA_DEVICE` memory types in [`base/src/declarative/ModuleRegistrations.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/declarative/ModuleRegistrations.cpp). This registration serves as the single source of truth for the analyzer:

```cpp
// In ModuleRegistrations.cpp
ModuleRegistrationBuilder<MyCustomCuda>("MyCustomCuda")
    .inputPin("input", FrameType::RAW_IMAGE, FrameMetadata::CUDA_DEVICE)
    .outputPin("output", FrameType::RAW_IMAGE, FrameMetadata::CUDA_DEVICE)
    .tag("function:custom")
    .tag("backend:cuda");

```

Once registered, any pipeline connecting a CPU-only source to `MyCustomCuda` automatically receives the necessary `CudaMemCopy` bridge. For manual transfers in custom preprocessing steps, explicitly insert a `CudaMemCopy` module in the JSON configuration:

```json
{
    "modules": {
        "upload": { "type": "CudaMemCopy", "props": { "kind": "HostToDevice" } },
        "process": { "type": "MyCustomCuda" }
    },
    "connections": [
        { "from": "source", "to": "upload" },
        { "from": "upload", "to": "process" }
    ]
}

```

## Detecting and Preventing Memory Leaks

The test harness tracks GPU allocations via the driver's reference counting. Run with `--detect_memory_leaks=0` only for debugging specific allocation patterns. For production code, ensure every `cuDevicePrimaryCtxRetain` has a matching `cuDevicePrimaryCtxRelease` in the destructor.

Query the module registry to find GPU-accelerated alternatives and avoid suboptimal CPU-only modules:

```cpp
auto gpuResizes = registry.getModulesWithTags(
    {"function:resize", "backend:cuda"}); // Returns ["ResizeNPPI"]
if (!gpuResizes.empty()) {
    // Use GPU variant to avoid Host→Device transfers
}

```

## Summary

- **Declare explicit `memType`** on every PIN in [`ModuleRegistrations.cpp`](https://github.com/apra-labs/aprapipes/blob/main/ModuleRegistrations.cpp) to enable automatic validation and bridge insertion.
- **Use `HOST_PINNED` allocations** via `apra_cudamallochost_allocator` for all host-side buffers that feed GPU modules.
- **Retain the primary CUDA context** using `cuDevicePrimaryCtxRetain` rather than `cuCtxCreate` to prevent per-instance memory leaks.
- **Leverage the shared CUDA stream** from `ModuleFactory` instead of creating custom streams for standard processing.
- **Allow the `PipelineAnalyzer` to insert bridges** automatically rather than manually managing `CudaMemCopy` modules unless specifically required.
- **Explicitly release all CUDA contexts** in destructors to prevent accumulation during repeated module instantiation in tests.

## Frequently Asked Questions

### How does ApraPipes automatically manage GPU memory transfers?

ApraPipes uses a per-PIN memory-type model where each module declares its input and output memory locations (HOST, HOST_PINNED, CUDA_DEVICE, or DMABUF) via `PinDef::memType` in [`Metadata.h`](https://github.com/apra-labs/aprapipes/blob/main/Metadata.h). The `PipelineAnalyzer` inspects these declarations and automatically inserts `CudaMemCopy` bridges when connecting modules with mismatched memory types, ensuring data reaches the GPU without manual intervention while keeping frames on the device across compatible modules.

### What is the correct way to allocate host memory for GPU uploads in ApraPipes?

Use the `apra_cudamallochost_allocator` class defined in `base/src/apra_cudamallochost_allocator.cu` to allocate page-locked (pinned) host memory. Pinned memory enables asynchronous DMA transfers and maximizes PCIe bandwidth for host-to-device copies. Allocate with `allocator.malloc(size)`, use the buffer to fill frame data, and the framework will automatically bridge the pinned memory to CUDA modules.

### Why should I use cuDevicePrimaryCtxRetain instead of cuCtxCreate?

Using `cuDevicePrimaryCtxRetain` (as implemented in [`base/src/H264DecoderNvCodecHelper.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/H264DecoderNvCodecHelper.cpp)) prevents per-instance GPU memory leaks that occur when modules repeatedly create and destroy contexts. The primary context is shared across the application and managed by the CUDA driver, eliminating OOM failures in long-running test suites and production pipelines that dynamically create decoder or encoder instances.

### How do I register a new CUDA module to work with the automatic bridge system?

Register the module in [`base/src/declarative/ModuleRegistrations.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/declarative/ModuleRegistrations.cpp) using `ModuleRegistrationBuilder`, explicitly setting `FrameMetadata::CUDA_DEVICE` for both input and output pins. This declaration allows the `PipelineAnalyzer` to detect when your module requires GPU memory and automatically insert the necessary `CudaMemCopy` bridges from upstream HOST or HOST_PINNED sources.