# How the ArmorPaint Layer System Manages Compositing, Blending Modes, and Stencil Operations

> Explore ArmorPaint's layer system managing compositing, blending, and stencil operations. Discover how util_layer.c, tab_layers.c, and util_stencil.c work together for real-time texture control.

- Repository: [Armory 3D/armorpaint](https://github.com/armory3d/armorpaint)
- Tags: internals
- Published: 2026-09-14

---

**ArmorPaint’s layer system orchestrates real-time texture compositing through three specialized source files: [`tab_layers.c`](https://github.com/armory3d/armorpaint/blob/main/tab_layers.c) drives the UI and writes blend modes to the `slot_layer_t` structure, [`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c) selects GPU pipelines based on those values during merging, and [`util_stencil.c`](https://github.com/armory3d/armorpaint/blob/main/util_stencil.c) transforms interactive stencil masks that modulate brush alpha before blending occurs.**

The **ArmorPaint layer system** relies on a tight coordination between UI state, GPU pipeline selection, and real-time stencil transformation. Understanding how [`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c) and [`tab_layers.c`](https://github.com/armory3d/armorpaint/blob/main/tab_layers.c) interact to handle compositing—while [`util_stencil.c`](https://github.com/armory3d/armorpaint/blob/main/util_stencil.c) manages masking—reveals the engine’s approach to non-destructive texture painting.

## Core Architecture of the Layer System

The layer implementation spans three critical files in the Armory3D ArmorPaint repository:

- **[`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c)** – Contains the core compositing logic, layer flattening, and GPU pipeline dispatch for blend modes.
- **[`tab_layers.c`](https://github.com/armory3d/armorpaint/blob/main/tab_layers.c)** – Implements the layer panel UI where artists select blending modes and toggle stencil usage, writing values into layer slots.
- **[`util_stencil.c`](https://github.com/armory3d/armorpaint/blob/main/util_stencil.c)** – Handles the interactive transformation (translation, rotation, scaling) of brush stencil textures during painting operations.

Each file operates on a shared data structure called `slot_layer_t`, which stores texture references and blending parameters that persist across UI updates and GPU operations.

## Layer Data Structures and UI Control

Every layer in ArmorPaint is represented by a **`slot_layer_t`** structure containing:

- `texpaint` – RGBA32 color texture
- `texpaint_nor` – Normal map texture
- `texpaint_pack` – Packed material channels (occlusion, roughness, metallic, height)
- `mask_opacity` – Opacity value for mask layers
- `blending` – Integer identifier for the blend mode (e.g., 0 for normal, 1 for multiply, 2 for screen)

When an artist selects a blend mode from the drop-down in the Layers panel, **[`tab_layers.c`](https://github.com/armory3d/armorpaint/blob/main/tab_layers.c)** writes the selected integer into the `blending` field of the target layer’s `slot_layer_t` instance. This UI code creates the layer node and binds the blend mode value:

```c
// In the layer properties panel (tab_layers.c)
ui_node_t *layer_node = nodes_material_create_node("LAYER", NULL);
layer_node->buttons->buffer[BLEND_BUTTON]->default_value = i32_array_create_x(selected_blend_mode);

```

The UI does not perform any compositing itself; it merely updates the state that [`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c) will read during the next merge or flatten operation.

## GPU Pipeline Selection in util_layer.c

The **`layers_merge_layer`** function in [`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c) (lines 73–84) implements the compositing engine. When merging a source layer into a destination, the function inspects the `blending` value and configures the GPU pipeline accordingly:

```c
if (slot_layer_is_layer(l1)) {                     // Source layer
    if (l1->paint_base) {
        gpu_set_pipeline(pipes_merge);            // Normal color merge
        gpu_set_int(pipes_blending, slot_layer_get_blending(l1));
    }
    // ...
}

```

**Key implementation details:**

- **`pipes_merge`** – The GPU pipeline that performs the actual color blending operation
- **`pipes_blending`** – Uniform that receives the integer blend mode identifier (normal, multiply, screen, add, overlay, etc.)
- **Lines 100–108** – Handle mask layers using the same pipeline but with fixed `BLEND_TYPE_MIX` and opacity controlled via `pipes_opac_merge_mask`

The GPU shaders interpret the integer passed to `pipes_blending` to execute the corresponding mathematical blend operation between layer textures.

## Stencil Transformation and Masking via util_stencil.c

While [`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c) handles layer-to-layer compositing, **[`util_stencil.c`](https://github.com/armory3d/armorpaint/blob/main/util_stencil.c)** manages the brush stencil system that masks individual painting operations. The **`util_stencil_transform`** function (lines 4–52) runs every frame to update the stencil’s interactive transformation:

```c
void util_stencil_transform() {
    if (g_context->brush_stencil_image && keymap_shortcut(...)) {
        // Hit-test corners → scaling
        // Hit-test center → rotation
        // Otherwise → translation
        
        // Maintain center point after scaling
        g_context->brush_stencil_x += (old_w - new_w) / (float)base_w() / 2.0;
        g_context->brush_stencil_y += (old_h - new_h) / (float)base_h() / 2.0;
    }
}

```

During the painting pass, the transformed stencil texture (`g_context->brush_stencil_image`) is bound to the shader uniform `brush_stencil`. The brush shader samples the stencil’s greyscale values and multiplies them against the brush alpha, effectively masking the paint input before it reaches the layer’s color texture.

## Complete Data Flow Example

The following example demonstrates how all three components coordinate during a typical painting session:

```c
// 1. User selects "Multiply" blend mode in UI (tab_layers.c)
//    This writes value 1 to top_layer->blending

// 2. During layer merge, util_layer.c selects the pipeline
slot_layer_t *base = g_project->_->layers->buffer[0];
slot_layer_t *top  = g_project->_->layers->buffer[1];
layers_merge_layer(base, top, false);  
// Calls gpu_set_int(pipes_blending, 1) for multiply blending

// 3. User activates a stencil image
g_context->brush_stencil_image = my_stencil_texture;

// 4. Every frame during painting, util_stencil_transform updates position
util_stencil_transform();  // Updates brush_stencil_x, brush_stencil_y, scale, angle

// 5. Brush shader samples the transformed stencil and composites
//    float stencil_mask = texture(brush_stencil, uv).r;
//    float final_alpha = brush_alpha * stencil_mask;

```

## Summary

- **[`tab_layers.c`](https://github.com/armory3d/armorpaint/blob/main/tab_layers.c)** provides the interface for selecting blend modes and enabling stencils, persisting choices in `slot_layer_t` structures.
- **[`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c)** executes compositing through the `layers_merge_layer` function, mapping blend mode integers to GPU pipeline configurations (`pipes_merge`, `pipes_blending`).
- **[`util_stencil.c`](https://github.com/armory3d/armorpaint/blob/main/util_stencil.c)** supplies real-time stencil transformation via `util_stencil_transform`, allowing artists to position, scale, and rotate masks that filter brush input before layer blending occurs.
- The system separates UI state management from GPU execution, with `slot_layer_t` serving as the authoritative data structure linking user input to rendering operations.

## Frequently Asked Questions

### How does ArmorPaint store the selected blend mode for each layer?

The blend mode is stored as an integer in the `blending` field of the `slot_layer_t` structure. When a user selects a mode from the Layers panel, [`tab_layers.c`](https://github.com/armory3d/armorpaint/blob/main/tab_layers.c) writes the corresponding value (e.g., 0 for normal, 1 for multiply) to this field. During compositing, [`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c) reads this value via `slot_layer_get_blending()` and passes it to the GPU pipeline through the `pipes_blending` uniform.

### What function handles the actual merging of layers in ArmorPaint?

The **`layers_merge_layer`** function in [`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c) (lines 73–84) handles layer merging. It checks if the source layer is a paint layer or mask, selects the appropriate GPU pipeline (`pipes_merge` or `pipes_merge_mask`), and configures blend mode parameters before dispatching the draw call.

### How does the stencil system interact with the layer compositing pipeline?

The stencil system operates at the brush input stage rather than during layer merging. [`util_stencil.c`](https://github.com/armory3d/armorpaint/blob/main/util_stencil.c) transforms the stencil texture interactively, and the brush shader samples this texture to modulate paint alpha. This masked paint is then written to the layer’s `texpaint` texture. When layers are later merged via [`util_layer.c`](https://github.com/armory3d/armorpaint/blob/main/util_layer.c), the stencil-masked content is composited using the layer’s assigned blend mode.

### Where is the interactive stencil transformation logic implemented?

The interactive transformation logic resides in **[`util_stencil.c`](https://github.com/armory3d/armorpaint/blob/main/util_stencil.c)** within the `util_stencil_transform` function (lines 4–52). This code detects mouse interactions to distinguish between translation, scaling (corner hit-tests), and rotation (center hit-tests), updating the global brush stencil position, scale, and angle variables each frame.