Brush Engine Architecture in ArmorPaint: How `util_brush.c` Integrates with `util_raycast.c`

The brush engine in ArmorPaint decouples input handling in util_brush.c from geometric picking in util_raycast.c, using the former to drive per-frame brush lifecycle events and the latter to supply axis-aligned bounding box raycasting that determines whether painting occurs on specific objects.

The armory3d/armorpaint repository implements a modular brush system that separates user input detection from scene intersection testing. This brush engine architecture allows the painting pipeline to remain agnostic of mesh geometry while still supporting precise 3D viewport workflows. At the core of this design are three tightly coupled components: the brush update logic in paint/sources/util/util_brush.c, the node-based output processing in paint/sources/nodes_brush/brush_output_node.c, and the raycasting helpers in paint/sources/util/util_raycast.c.

Brush Input Handling and State Management in util_brush.c

The file util_brush.c serves as the primary entry point for the brush engine, orchestrating the per-frame lifecycle of brush strokes. The function util_brush_update() aggregates input from keyboard shortcuts, mouse states, and pen devices to determine if a painting operation should commence.

Input detection evaluates multiple trigger conditions simultaneously:

bool paint_down = keymap_shortcut(any_map_get(g_keymap, "action_paint"), SHORTCUT_TYPE_DOWN);
bool down = paint_down || decal_mask || set_clone_source ||
            keymap_shortcut(string_tmp("%s+%s", any_map_get(g_keymap, "brush_ruler"),
            any_map_get(g_keymap, "action_paint")), SHORTCUT_TYPE_DOWN) ||
            (pen_down("tip") && !keyboard_down("alt"));

When the Clone tool is active, the system monitors for the "set clone source" gesture. Upon detection, coordinates are stored and the flag util_brush_clone_source_down is raised, temporarily suspending normal painting operations to capture the source texture location.

Brush timing logic tracks the duration of the stroke through g_context->brush_time. While the brush remains active (down == true), the timer increments and brush_output_node_run() executes. Upon release, the engine resets the timer, sets dirty flags (brush_blend_dirty, layer_preview_dirty), and triggers tool-specific post-stroke actions such as color-ID fill updates.

Node-Based Brush Output Processing

The actual painting logic resides in brush_output_node.c, where brush_output_node_run() and brush_output_node_parse_inputs() transform raw input into GPU draw commands. This node-based architecture allows artists to chain arbitrary image processing nodes (masks, stencils, random generators) before final pixel application.

The parser extracts seven critical parameters from the node graph:

• Position (input0) → g_context->paint_vec
• Radius (input1) → g_context->brush_nodes_radius
• Scale (input2) → g_context->brush_nodes_scale
• Angle (input3) → g_context->brush_nodes_angle
• Opacity/Mask (input4) → texture or constant value
• Hardness (input5) → g_context->brush_nodes_hardness
• Stencil (input6) → stencil texture reference

Before emitting a stroke, the node implements painting guardrails that validate:

• Viewport boundary constraints (left/right/top/bottom limits)
• Layer state (fill layers, groups, or hidden layers block painting)
• UI interaction states (hovered widgets or active toolbars suspend strokes)
• Brush lock status

If validation passes, the node updates dirty flags (pdirty, ddirty), enables undo tracking via sculpt_push_undo(), and prepares the render pipeline to sample brush masks and stencils. The actual GPU draw occurs asynchronously within the render subsystem.

Geometric Picking with util_raycast.c

The util_raycast.c file provides lightweight geometric utilities for mouse-based object selection, operating independently from the brush pipeline but feeding critical data into it. Two functions comprise the core API:

• raycast_aabb_mouse(object_t *object) — Casts a ray from the current mouse coordinates (mouse_x, mouse_y) through the scene camera, returning the intersection point with the object's axis-aligned bounding box (AABB).
• point_in_aabb(object_t *object, vec4_t point) — Tests whether a world-space point lies within the object's AABB.

These helpers support higher-level tools that must resolve viewport coordinates into 3D surface positions. The Cursor tool, for example, relies on these functions to locate objects under the mouse before the brush system evaluates whether to paint or perform selection operations.

// Cursor tool implementation pattern
object_t *obj = scene_ray_cast(mouse_x, mouse_y, scene_camera);
if (obj) {
    vec4_t hit = raycast_aabb_mouse(obj);   // util_raycast.c
    // Use hit position for color sampling or cursor placement
}

Interaction Between Raycasting and Brush Execution

The separation between util_brush.c and util_raycast.c creates a clean data flow where geometry queries inform but do not control the painting pipeline. Raycasting results influence brush execution through the picking_object boolean checked inside brush_output_node_run().

When the active tool is TOOL_TYPE_CURSOR, the boolean picking_object evaluates to true, causing the brush node to abort painting and defer to the cursor tool's hit-handling logic. This architectural boundary ensures that util_brush.c processes only input state and timing, while util_raycast.c handles only geometric intersection, with brush_output_node.c mediating between them.

The typical per-frame execution cycle demonstrates this modularity:

void main_update() {
    // 1. Update brush state (detect press/release, manage timers)
    util_brush_update();
    
    // 2. Node output prepares paint vectors and validates constraints
    // 3. Raycasting (if cursor tool active) determines picking_object state
    // 4. Render pipeline executes GPU draw for valid strokes
}

The paint vector (g_context->paint_vec) computed by raycasting or other viewport projection methods passes into the brush engine as a normalized coordinate. This abstraction allows the same util_brush.c logic to function across 2-D UV canvases, 3-D meshes, or custom projection surfaces without modification to the core timing or input handling code.

Summary

• util_brush.c manages the per-frame brush lifecycle, input aggregation, clone-source gestures, and stroke timing via util_brush_update().
• brush_output_node.c implements the node-based painting logic, parsing radius, opacity, hardness, and stencil inputs while enforcing layer and UI constraints.
• util_raycast.c provides AABB raycasting utilities (raycast_aabb_mouse(), point_in_aabb()) for object picking without direct brush pipeline dependency.
• The Cursor tool bridges both systems by setting picking_object = true based on raycasting results, causing the brush node to skip painting and handle selection instead.
• The architecture enforces separation of concerns: input handling, geometric queries, and paint execution remain distinct modules communicating through g_context state.

Frequently Asked Questions

What is the primary responsibility of util_brush_update() in ArmorPaint?

util_brush_update() serves as the main entry point for brush input processing in util_brush.c. It evaluates keyboard shortcuts, mouse states, and pen inputs to determine if a brush stroke should start, continue, or end, while managing the clone-source state and incrementing g_context->brush_time for active strokes.

How does util_raycast.c influence whether painting occurs in the viewport?

While util_raycast.c does not directly modify brush state, its raycast_aabb_mouse() function supplies geometric intersection data used by tools like the Cursor tool. When the Cursor tool detects an object hit via raycasting, it sets picking_object = true inside brush_output_node_run(), which causes the brush engine to skip painting and perform selection or color sampling instead.

What specific inputs does brush_output_node_parse_inputs() extract from the node graph?

The parser extracts seven parameters: position (mapped to g_context->paint_vec), radius, scale, angle, opacity/mask (as texture or constant), hardness, and stencil texture. These values flow from the node graph into global context variables that the render pipeline uses to configure the brush stamp.

Why does the brush engine separate util_brush.c from util_raycast.c?

This separation allows the brush timing and input logic to remain geometry-agnostic. By isolating AABB raycasting in util_raycast.c, the system can reuse util_brush_update() for 2-D UV painting, 3-D mesh painting, or custom projection modes without modifying the core input handling code, while raycasting utilities remain available to any tool requiring viewport picking.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}