Understanding ncnn's Packing Layout Mechanism and ARM SIMD Optimizations with use_packing_layout

ncnn's packing layout mechanism groups scalar tensor elements into vector-compatible blocks (elempack) to enable ARM NEON SIMD instructions to process multiple values simultaneously, delivering significant speedups when the use_packing_layout option is enabled.

The Tencent/ncnn inference framework employs a sophisticated memory layout strategy to maximize throughput on ARM processors. By transforming traditional scalar tensors into packed formats, ncnn allows convolution and fully-connected layers to leverage 128-bit and 256-bit SIMD registers efficiently. This deep dive explores the implementation from the global use_packing_layout flag down to the NEON intrinsics that perform the actual data transposition.

What Is the ncnn Packing Layout?

In ncnn, elempack defines how many scalar values are stored in a single logical element. When elempack equals 1, data follows a standard scalar layout. When packing is active, elempack becomes 4, 8, or 16, grouping that many floats (or other data types) into contiguous memory blocks.

This packing enables SIMD units to load, compute, and store multiple values in single instructions. For example, with elempack = 4 on ARM, a vld1q_f32 instruction loads four floats at once into a 128-bit NEON register.

The global switch controlling this behavior resides in src/option.h at lines 98-102, where the use_packing_layout boolean defaults to true. When enabled, the framework requests that layers produce and consume packed tensors; when disabled, the network operates entirely in scalar mode for maximum compatibility.

Where Packing Happens: From Generic to ARM-Specific

The Base Packing Layer

The generic implementation lives in src/layer/packing.cpp (lines 25-45). This base class handles metadata updates—adjusting dims, w, h, cstep, and elemsize—without necessarily touching the raw byte layout when the transition is trivial. It serves as the fallback for unsupported architectures or unusual packing conversions.

ARM NEON Specialization

For ARM devices, src/layer/arm/packing_arm.cpp overrides the forward pass with high-performance NEON implementations (lines 24-66). This file contains the specialized logic for converting between scalar and packed layouts using SIMD intrinsics. The code detects the tensor's bit-width (elembits) to dispatch appropriate routines for 8-bit integers, 16-bit bfloat16/float16, or 32-bit float32 data.

ARM SIMD Optimization Mechanics

The ARM-specific packing implementation follows a precise pipeline to maximize throughput:

Element-Size Detection – The code first queries elembits to determine if the tensor contains 8-bit, 16-bit, or 32-bit data, routing execution to specialized handlers for each width.

Packing Direction Flags – Boolean variables determine the conversion pattern:

bool pack1to4 = elempack == 1 && out_elempack == 4;
bool pack4to1 = elempack == 4 && out_elempack == 1;

Similar flags exist for 8-element packs. These conditions trigger different NEON code paths in packing_arm.cpp (lines 59-66).

Vectorized Transposition – When packing from 1→4 elements, the implementation loads four consecutive rows using vld1q_f32, constructs a float32x4x4_t structure, and stores the transposed result with vst4q_f32. The reverse 4→1 operation uses vld4q_f32 followed by separate stores. This pattern eliminates scalar copies and achieves full-width NEON transfers per iteration (lines 112-138).

Higher-Order Packs – For 8-element packs used with half-precision (fp16) or bfloat16 tensors, the code employs vld1q_u16 to load eight 16-bit values, then uses vzipq_u16 and vtrn instructions (or inline assembly for 8×8 transposes) to reorganize data in registers before storing. This transpose-free conversion is critical for performance when handling large feature maps (lines 170-225).

Framework Integration and use_packing_layout

The packing mechanism integrates deeply into ncnn's graph construction in src/net.cpp (line 234). During network initialization, ncnn checks each layer's support_packing flag alongside the global Option::use_packing_layout setting. When both conditions are true, the framework automatically inserts implicit Packing layers at the boundaries of operators that support vectorized execution.

Weight loading also respects these flags. If a layer supports packing and the option is enabled, weight blobs are stored in packed form using opt_download.use_packing_layout = layer->support_packing, eliminating runtime repacking overhead.

The test suite in tests/test_packing.cpp (lines 48-155) validates both packed and scalar execution paths by toggling opt.use_packing_layout and verifying numerical consistency across modes.

When to Disable Packing

While packing improves performance on modern ARM cores, certain scenarios warrant disabling the feature:

• Numerical Debugging – Scalar mode eliminates transposition operations, making it easier to trace individual element values and identify precision issues.
• Legacy Hardware – Older ARM cores without NEON support automatically fall back based on runtime queries of support_fp16_storage, but explicitly disabling packing ensures compatibility.
• Memory Constraints – Packed tensors for 8-bit data occupy elempack times more memory per logical element (four int8 values packed together consume four bytes instead of one), potentially increasing peak memory usage in quantized models.

Practical Code Examples

Configuring use_packing_layout at Runtime

Enable or disable packing when loading a model to control SIMD optimization levels:

#include <net.h>

int main()
{
    ncnn::Net net;
    
    // Modify the default options
    ncnn::Option opt = net.opt;
    opt.use_packing_layout = true;  // Enable ARM NEON packing (default)
    // opt.use_packing_layout = false; // Force scalar mode for debugging
    net.opt = opt;
    
    // Load model weights
    net.load_param("model.param");
    net.load_model("model.bin");
    
    // Inference automatically respects the packing setting
    ncnn::Extractor ex = net.create_extractor();
    ncnn::Mat in = ncnn::Mat::from_pixels(image_data, ncnn::Mat::PIXEL_BGR, w, h);
    ex.input("data", in);
    
    ncnn::Mat out;
    ex.extract("prob", out);
    
    return 0;
}

Querying Tensor Packing Status

Inspect any blob to determine its current layout:

ncnn::Mat blob = ...;  // Output from any layer
printf("dims=%d w=%d h=%d c=%d pack=%d elemsize=%zu\n",
    blob.dims, blob.w, blob.h, blob.c,
    blob.elempack, blob.elemsize);

Typical packed output on ARM appears as: dims=3 w=28 h=28 c=64 pack=4 elemsize=16, indicating four floats (16 bytes) per packed element.

Manual Packing Layer Insertion

For custom network architectures, explicitly insert packing layers to control data layout:

// Create packing layer to convert scalar to 4-packed
ncnn::Packing* pack = static_cast<ncnn::Packing*>(net.create_layer("Packing"));
pack->out_elempack = 4;  // Target 4-element vectors

// Configure input/output blobs (pseudo-code)
pack->bottom_blob = input_blob;
pack->forward_inplace(input_blob);

// Now input_blob contains packed data suitable for vectorized convolution

Summary

• Packing layout in ncnn groups scalar values into vectors (elempack = 4, 8, or 16) to exploit ARM NEON SIMD parallelism.
• The use_packing_layout option in src/option.h globally controls whether the network operates in packed or scalar mode.
• ARM optimizations reside in src/layer/arm/packing_arm.cpp, using intrinsics like vld4q_f32 and vst4q_f32 for high-throughput data transposition.
• The framework automatically inserts packing layers during graph construction in src/net.cpp based on layer capabilities and global options.
• Disabling packing aids debugging and reduces memory overhead for 8-bit quantized models at the cost of SIMD performance.

Frequently Asked Questions

What does elempack mean in ncnn?

elempack indicates the number of scalar values contained in each logical element of a tensor. When elempack equals 4, each "element" actually stores four scalar values (e.g., four floats) in contiguous memory. This allows ARM NEON instructions to load an entire vector with one vld1q_f32 instruction. The value is typically 1 (scalar), 4 (standard float32 packing), or 8 (for fp16/bf16 on 128-bit architectures).

How do I disable packing layout for debugging?

Set use_packing_layout = false in the network options before loading the model. This forces all layers to operate on scalar tensors (elempack = 1), eliminating transposition operations and making it easier to compare intermediate outputs against reference implementations or other frameworks.

Why is ARM NEON faster with packed tensors?

Packed tensors align memory access patterns with NEON's 128-bit register width. When elempack = 4, loading four consecutive floats fills a float32x4_t register exactly. Without packing, scalar loops process one element per iteration, utilizing only 25% of the SIMD register width and incurring significant instruction overhead. The packing_arm.cpp implementation uses interleaved loads (vld4q_f32) that perform the necessary matrix transpositions in registers without additional memory passes.

Does packing layout increase memory usage?

For 32-bit float data, packing does not increase memory consumption because four floats occupy 16 bytes whether stored as four scalars or one packed element. However, for 8-bit integers, packing four values together effectively quadruples the memory footprint per logical element because ncnn pads packed elements to maintain alignment. In memory-constrained environments running int8 quantization, disabling use_packing_layout may reduce peak RAM usage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}