How the ArmorPaint Layer System Manages Compositing, Blending Modes, and Stencil Operations

ArmorPaint’s layer system orchestrates real-time texture compositing through three specialized source files: tab_layers.c drives the UI and writes blend modes to the slot_layer_t structure, util_layer.c selects GPU pipelines based on those values during merging, and util_stencil.c transforms interactive stencil masks that modulate brush alpha before blending occurs.

The ArmorPaint layer system relies on a tight coordination between UI state, GPU pipeline selection, and real-time stencil transformation. Understanding how util_layer.c and tab_layers.c interact to handle compositing—while util_stencil.c manages masking—reveals the engine’s approach to non-destructive texture painting.

Core Architecture of the Layer System

The layer implementation spans three critical files in the Armory3D ArmorPaint repository:

• util_layer.c – Contains the core compositing logic, layer flattening, and GPU pipeline dispatch for blend modes.
• tab_layers.c – Implements the layer panel UI where artists select blending modes and toggle stencil usage, writing values into layer slots.
• util_stencil.c – Handles the interactive transformation (translation, rotation, scaling) of brush stencil textures during painting operations.

Each file operates on a shared data structure called slot_layer_t, which stores texture references and blending parameters that persist across UI updates and GPU operations.

Layer Data Structures and UI Control

Every layer in ArmorPaint is represented by a slot_layer_t structure containing:

• texpaint – RGBA32 color texture
• texpaint_nor – Normal map texture
• texpaint_pack – Packed material channels (occlusion, roughness, metallic, height)
• mask_opacity – Opacity value for mask layers
• blending – Integer identifier for the blend mode (e.g., 0 for normal, 1 for multiply, 2 for screen)

When an artist selects a blend mode from the drop-down in the Layers panel, tab_layers.c writes the selected integer into the blending field of the target layer’s slot_layer_t instance. This UI code creates the layer node and binds the blend mode value:

// In the layer properties panel (tab_layers.c)
ui_node_t *layer_node = nodes_material_create_node("LAYER", NULL);
layer_node->buttons->buffer[BLEND_BUTTON]->default_value = i32_array_create_x(selected_blend_mode);

The UI does not perform any compositing itself; it merely updates the state that util_layer.c will read during the next merge or flatten operation.

GPU Pipeline Selection in util_layer.c

The layers_merge_layer function in util_layer.c (lines 73–84) implements the compositing engine. When merging a source layer into a destination, the function inspects the blending value and configures the GPU pipeline accordingly:

if (slot_layer_is_layer(l1)) {                     // Source layer
    if (l1->paint_base) {
        gpu_set_pipeline(pipes_merge);            // Normal color merge
        gpu_set_int(pipes_blending, slot_layer_get_blending(l1));
    }
    // ...
}

Key implementation details:

• pipes_merge – The GPU pipeline that performs the actual color blending operation
• pipes_blending – Uniform that receives the integer blend mode identifier (normal, multiply, screen, add, overlay, etc.)
• Lines 100–108 – Handle mask layers using the same pipeline but with fixed BLEND_TYPE_MIX and opacity controlled via pipes_opac_merge_mask

The GPU shaders interpret the integer passed to pipes_blending to execute the corresponding mathematical blend operation between layer textures.

Stencil Transformation and Masking via util_stencil.c

While util_layer.c handles layer-to-layer compositing, util_stencil.c manages the brush stencil system that masks individual painting operations. The util_stencil_transform function (lines 4–52) runs every frame to update the stencil’s interactive transformation:

void util_stencil_transform() {
    if (g_context->brush_stencil_image && keymap_shortcut(...)) {
        // Hit-test corners → scaling
        // Hit-test center → rotation
        // Otherwise → translation
        
        // Maintain center point after scaling
        g_context->brush_stencil_x += (old_w - new_w) / (float)base_w() / 2.0;
        g_context->brush_stencil_y += (old_h - new_h) / (float)base_h() / 2.0;
    }
}

During the painting pass, the transformed stencil texture (g_context->brush_stencil_image) is bound to the shader uniform brush_stencil. The brush shader samples the stencil’s greyscale values and multiplies them against the brush alpha, effectively masking the paint input before it reaches the layer’s color texture.

Complete Data Flow Example

The following example demonstrates how all three components coordinate during a typical painting session:

// 1. User selects "Multiply" blend mode in UI (tab_layers.c)
//    This writes value 1 to top_layer->blending

// 2. During layer merge, util_layer.c selects the pipeline
slot_layer_t *base = g_project->_->layers->buffer[0];
slot_layer_t *top  = g_project->_->layers->buffer[1];
layers_merge_layer(base, top, false);  
// Calls gpu_set_int(pipes_blending, 1) for multiply blending

// 3. User activates a stencil image
g_context->brush_stencil_image = my_stencil_texture;

// 4. Every frame during painting, util_stencil_transform updates position
util_stencil_transform();  // Updates brush_stencil_x, brush_stencil_y, scale, angle

// 5. Brush shader samples the transformed stencil and composites
//    float stencil_mask = texture(brush_stencil, uv).r;
//    float final_alpha = brush_alpha * stencil_mask;

Summary

• tab_layers.c provides the interface for selecting blend modes and enabling stencils, persisting choices in slot_layer_t structures.
• util_layer.c executes compositing through the layers_merge_layer function, mapping blend mode integers to GPU pipeline configurations (pipes_merge, pipes_blending).
• util_stencil.c supplies real-time stencil transformation via util_stencil_transform, allowing artists to position, scale, and rotate masks that filter brush input before layer blending occurs.
• The system separates UI state management from GPU execution, with slot_layer_t serving as the authoritative data structure linking user input to rendering operations.

Frequently Asked Questions

How does ArmorPaint store the selected blend mode for each layer?

The blend mode is stored as an integer in the blending field of the slot_layer_t structure. When a user selects a mode from the Layers panel, tab_layers.c writes the corresponding value (e.g., 0 for normal, 1 for multiply) to this field. During compositing, util_layer.c reads this value via slot_layer_get_blending() and passes it to the GPU pipeline through the pipes_blending uniform.

What function handles the actual merging of layers in ArmorPaint?

The layers_merge_layer function in util_layer.c (lines 73–84) handles layer merging. It checks if the source layer is a paint layer or mask, selects the appropriate GPU pipeline (pipes_merge or pipes_merge_mask), and configures blend mode parameters before dispatching the draw call.

How does the stencil system interact with the layer compositing pipeline?

The stencil system operates at the brush input stage rather than during layer merging. util_stencil.c transforms the stencil texture interactively, and the brush shader samples this texture to modulate paint alpha. This masked paint is then written to the layer’s texpaint texture. When layers are later merged via util_layer.c, the stencil-masked content is composited using the layer’s assigned blend mode.

Where is the interactive stencil transformation logic implemented?

The interactive transformation logic resides in util_stencil.c within the util_stencil_transform function (lines 4–52). This code detects mouse interactions to distinguish between translation, scaling (corner hit-tests), and rotation (center hit-tests), updating the global brush stencil position, scale, and angle variables each frame.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}