# ncnn Vulkan PipelineCache: How It Accelerates GPU Inference Performance

> Discover ncnn Vulkan PipelineCache for faster GPU inference. Learn how it stores compiled pipelines, cutting latency from milliseconds to microseconds. Accelerate your AI workloads now.

- Repository: [Tencent/ncnn](https://github.com/tencent/ncnn)
- Tags: deep-dive
- Published: 2026-02-23

---

**The ncnn Vulkan PipelineCache stores compiled compute pipelines keyed by shader bytecode and execution parameters, eliminating expensive recompilation overhead and reducing inference latency from milliseconds to microseconds on subsequent runs.**

The ncnn inference framework leverages Vulkan compute shaders to accelerate deep learning workloads on GPUs. At the heart of this optimization lies the **Vulkan PipelineCache**, a mechanism that persists expensive pipeline creation artifacts across inference sessions. By caching compiled shader modules and pipeline objects, ncnn avoids the substantial driver overhead associated with SPIR-V compilation and pipeline validation.

## What Is the ncnn Vulkan PipelineCache?

The Vulkan PipelineCache in ncnn is a software layer built atop the native Vulkan `VkPipelineCache` object. It maintains a mapping between unique pipeline configurations and their corresponding compiled Vulkan artifacts. When ncnn needs to execute a compute shader, it queries this cache to retrieve previously created `VkShaderModule`, `VkPipelineLayout`, `VkDescriptorSetLayout`, and `VkPipeline` objects.

According to the source code in [`src/pipelinecache.cpp`](https://github.com/Tencent/ncnn/blob/main/src/pipelinecache.cpp), the cache stores these artifacts in `PipelineCachePrivate::cache_digests` and `PipelineCachePrivate::cache_artifacts`. When a cache hit occurs, ncnn returns the stored objects instantly, bypassing the expensive `vkCreateShaderModule` and `vkCreateComputePipelines` calls.

## How the PipelineCache Works

### Cache Key Generation

Each compute pipeline is uniquely identified by a **digest** calculated in `PipelineCachePrivate::pipeline_cache_digest` (located in [`src/pipelinecache.cpp`](https://github.com/Tencent/ncnn/blob/main/src/pipelinecache.cpp) lines 54-90). This digest function combines multiple parameters to create a SHA-256 hash that serves as the cache key:

- The SPIR-V bytecode of the shader module
- Specialization constants passed to the shader
- Local work-group dimensions (`local_size_x`, `local_size_y`, `local_size_z`)
- Subgroup size configuration
- Inference precision flags (FP16/INT8 settings)

This comprehensive keying ensures that pipelines with different optimization settings or shader variants are cached separately, preventing incorrect shader reuse while maximizing valid cache hits.

### Storage and Lookup Mechanism

The cache maintains two parallel vectors in `PipelineCachePrivate`: `cache_digests` stores the SHA-256 hashes, while `cache_artifacts` stores the corresponding `PipelineCachePrivate::CacheArtifact` structures containing the Vulkan handles.

When `PipelineCache::get_pipeline` is invoked, it calculates the digest for the requested configuration and scans `cache_digests` for a match. On a cache hit, the function returns the stored `VkShaderModule`, `VkPipelineLayout`, `VkPipeline`, and descriptor-related objects. On a miss, the cache proceeds to create these objects using `vkCreateShaderModule` and `vkCreateComputePipelines`, then stores the results for future queries.

## PipelineCache Integration in ncnn

### Device-Level Cache Creation

The `VulkanDevice` class manages the lifecycle of the pipeline cache. In [`src/gpu.cpp`](https://github.com/Tencent/ncnn/blob/main/src/gpu.cpp) (around line 2400), the device initialization code calls `vkCreatePipelineCache` to create a native Vulkan pipeline cache object. This device-level cache provides persistence across multiple `ncnn::Net` instances running on the same GPU.

The `VulkanDevice` exposes this cache through `VulkanDevice::get_pipeline_cache()`, declared in [`src/gpu.h`](https://github.com/Tencent/ncnn/blob/main/src/gpu.h) (line 468), allowing pipeline creation code to access the cache during shader compilation.

### Pipeline Creation Flow

Every compute pipeline in ncnn flows through the `Pipeline` class defined in [`src/pipeline.cpp`](https://github.com/Tencent/ncnn/blob/main/src/pipeline.cpp). The `Pipeline::create` method (lines 21-26) retrieves the active pipeline cache and calls `PipelineCache::get_pipeline`:

```cpp
// src/pipeline.cpp (simplified)
int Pipeline::create(const uint32_t* spv_data, size_t spv_data_size,
                     const std::vector<vk_specialization_type>& specializations)
{
    const PipelineCache* pipeline_cache = vkdev->get_pipeline_cache();

    // Query cache or create new pipeline
    return pipeline_cache->get_pipeline(spv_data, spv_data_size, specializations,
                                       d->local_size_x, d->local_size_y, d->local_size_z,
                                       d->subgroup_size,
                                       &d->shader_module, &d->descriptorset_layout,
                                       &d->pipeline_layout, &d->pipeline,
                                       &d->descriptor_update_template,
                                       d->shader_info);
}

```

This integration ensures that every shader compilation benefits from caching without requiring manual intervention from ncnn users.

### Custom Cache Usage

Advanced users can supply their own `PipelineCache` instance through the `Option` structure. By setting `net.opt.pipeline_cache` to a custom cache object, developers can share cached pipelines across multiple network instances or serialize the cache to disk for faster application startup:

```cpp
// Create a custom PipelineCache
ncnn::PipelineCache* my_cache = new ncnn::PipelineCache(vkdev);

// Attach to network options
ncnn::Net net;
net.opt.use_vulkan_compute = true;
net.opt.pipeline_cache = my_cache;

// Load model - pipelines will be cached in my_cache
net.load_param("model.param");
net.load_model("model.bin");

```

When no custom cache is provided, ncnn automatically falls back to the device-level cache via `VulkanDevice::get_pipeline_cache()`.

## Performance Impact of Vulkan Pipeline Caching

The Vulkan PipelineCache delivers substantial performance improvements by eliminating redundant compilation overhead. Creating a compute pipeline from SPIR-V bytecode involves multiple expensive driver operations: shader module creation, descriptor set layout validation, pipeline layout construction, and final pipeline compilation.

On desktop GPUs, this compilation process typically requires several milliseconds per unique shader. On mobile GPUs, the cost escalates to tens of milliseconds due to more constrained driver implementations and thermal considerations. For deep learning models containing dozens or hundreds of distinct compute shaders, these compilation times accumulate into significant startup latency.

The PipelineCache reduces this overhead to microseconds on cache hits. Once a pipeline is compiled and stored, subsequent retrievals require only a hash lookup and handle assignment. This optimization proves particularly valuable for:
- **Batch inference scenarios** where the same model processes multiple inputs sequentially
- **Real-time applications** requiring consistent frame-to-frame latency
- **Multi-model deployments** sharing common shader kernels across different networks

The first inference still incurs compilation costs for uncached pipelines, but steady-state performance improves dramatically as the cache population grows.

## Implementation Details and Source Files

The Vulkan PipelineCache implementation spans several key files in the ncnn repository:

| File | Purpose |
|------|---------|
| [`src/pipelinecache.h`](https://github.com/Tencent/ncnn/blob/main/src/pipelinecache.h) | Public API declaration including `PipelineCache` class, `clear()` method, and `get_pipeline` signature |
| [`src/pipelinecache.cpp`](https://github.com/Tencent/ncnn/blob/main/src/pipelinecache.cpp) | Core implementation containing `pipeline_cache_digest` (lines 54-90), cache storage vectors, and lookup logic |
| [`src/pipeline.cpp`](https://github.com/Tencent/ncnn/blob/main/src/pipeline.cpp) | High-level pipeline creation wrapper that queries the cache before invoking Vulkan creation functions |
| [`src/gpu.cpp`](https://github.com/Tencent/ncnn/blob/main/src/gpu.cpp) | Device initialization code around line 2400 that creates the native `VkPipelineCache` via `vkCreatePipelineCache` |
| [`src/gpu.h`](https://github.com/Tencent/ncnn/blob/main/src/gpu.h) | Declaration of `VulkanDevice::get_pipeline_cache()` at line 468, providing cache access to pipeline creation code |
| [`docs/how-to-use-and-FAQ/vulkan-notes.md`](https://github.com/Tencent/ncnn/blob/main/docs/how-to-use-and-FAQ/vulkan-notes.md) | User-facing documentation explaining Vulkan compute enablement and cache utilization |

These components work together to provide transparent pipeline caching that requires no explicit user configuration while offering advanced customization options for specialized deployment scenarios.

## Summary

- The **ncnn Vulkan PipelineCache** stores compiled compute pipelines to eliminate redundant shader compilation and pipeline creation overhead.
- Cache keys are generated using **SHA-256 digests** that combine SPIR-V bytecode, specialization constants, work-group sizes, and precision flags in [`src/pipelinecache.cpp`](https://github.com/Tencent/ncnn/blob/main/src/pipelinecache.cpp).
- The cache delivers **microsecond-level retrieval** versus millisecond-level compilation, dramatically improving steady-state inference performance.
- Users can leverage the **default device cache** automatically or supply **custom `PipelineCache` instances** via `Option::pipeline_cache` for cross-session persistence.
- Implementation spans [`src/pipelinecache.h`](https://github.com/Tencent/ncnn/blob/main/src/pipelinecache.h), [`src/pipelinecache.cpp`](https://github.com/Tencent/ncnn/blob/main/src/pipelinecache.cpp), [`src/pipeline.cpp`](https://github.com/Tencent/ncnn/blob/main/src/pipeline.cpp), and [`src/gpu.cpp`](https://github.com/Tencent/ncnn/blob/main/src/gpu.cpp), integrating seamlessly with ncnn's Vulkan compute backend.

## Frequently Asked Questions

### How does ncnn's PipelineCache differ from the native Vulkan pipeline cache?

The native Vulkan `VkPipelineCache` stores driver-specific binary data for pipeline objects, but ncnn's **PipelineCache** adds a higher-level abstraction that caches the entire creation process including shader modules, descriptor set layouts, and pipeline layouts. While the native cache helps with driver-level optimizations, ncnn's implementation in [`src/pipelinecache.cpp`](https://github.com/Tencent/ncnn/blob/main/src/pipelinecache.cpp) provides faster lookup by caching the complete set of Vulkan objects keyed by shader content, eliminating even the overhead of descriptor layout validation.

### Can I persist the PipelineCache across application restarts?

Yes, though it requires manual serialization. The `PipelineCache` class stores compiled pipelines in memory during runtime. To persist across restarts, you would need to extract the native Vulkan pipeline cache data using `vkGetPipelineCacheData` from the underlying `VkPipelineCache` object, save it to disk, and reload it on the next run. ncnn does not provide built-in serialization APIs for the PipelineCache, but advanced users can access the native Vulkan handles through the `VulkanDevice` to implement custom persistence layers.

### What happens if the PipelineCache runs out of memory?

The PipelineCache uses standard `std::vector` storage for digests and artifacts in `PipelineCachePrivate`. If memory becomes constrained, the cache will grow until system memory limits are reached, potentially causing allocation failures. However, ncnn provides the `PipelineCache::clear()` method (declared in [`src/pipelinecache.h`](https://github.com/Tencent/ncnn/blob/main/src/pipelinecache.h)) to manually evict all cached entries and reclaim memory. For long-running services processing many different models, periodic cache clearing or implementing a custom LRU eviction policy on top of the PipelineCache may be necessary to manage memory consumption.

### Does the PipelineCache work with all Vulkan-capable GPUs?

The PipelineCache functionality is available on all Vulkan-capable devices supported by ncnn, including mobile GPUs (ARM Mali, Qualcomm Adreno) and desktop GPUs (NVIDIA, AMD, Intel). The cache mechanism itself is hardware-agnostic, relying on standard Vulkan API calls. However, the performance benefits vary by platform—mobile GPUs typically see the most dramatic improvements because their drivers incur higher compilation overhead (tens of milliseconds per shader) compared to desktop drivers. The cache is automatically enabled when `use_vulkan_compute` is set to true in the ncnn options, requiring no platform-specific configuration.