How the Modly ForEachNode Handles Batch Processing of Assets: A Deep Dive

The Modly ForEachNode handles batch processing by pre-scanning folders to build an alphabetically-sorted file list, treating each file as a loop iteration, and optionally grouping multiple iterators that share downstream body nodes for lock-step execution.

The ForEachNode is the core iterator that drives batch processing in Modly's workflow engine. If you're working with collections of images, text files, or 3D meshes, understanding how this node manages iteration, progress tracking, and pause/resume functionality is essential for building reliable automation pipelines. This article examines the complete implementation across the UI layer and runtime engine in the lightningpixel/modly repository.

Architecture Overview: UI Component and Runtime Engine

The ForEachNode splits responsibilities between a React-based UI component and the core workflow execution store. This separation keeps the interface responsive while the engine handles file I/O and state management.

ForEachNode.tsx: Folder Selection and Progress Display

The UI component lives at src/areas/workflows/nodes/ForEachNode.tsx. Users interact with two primary controls:

• Folder picker (lines 41-61): Opens a native directory dialog via window.electron.fs.selectDirectory
• Asset mode dropdown (lines 62-87): Selects between image, text, or mesh processing

While a workflow runs, the UI locks these controls and displays iteration progress through a badge showing current/total (lines 102-135). Pause, Continue, and Retry buttons appear when the user requests a break at the next iteration boundary.

// UI – pick folder and show iteration progress
const pickDir = async () => {
  const picked = await window.electron.fs.selectDirectory(dir || undefined);
  if (picked) updateNodeData(id, { params: { ...data.params, dir: picked } });
};

return (
  <BaseNode …>
    <select … value={mode} onChange={e => updateNodeData(id, { params: { …data.params, mode: e.target.value as Mode, dir: undefined } })}>
      <option value="image">Image</option>
      <option value="text">Text</option>
      <option value="mesh">Mesh</option>
    </select>
    <button onClick={pickDir}>…{dirLabel}</button>
    {progress && <span>{progress.current}/{progress.total}</span>}
  </BaseNode>
);

workflowRunStore.ts: Core Batch Processing Engine

The runtime logic in src/areas/workflows/workflowRunStore.ts handles four critical phases: pre-listing files, building loop metadata, executing iterations, and managing flow control.

Phase 1: Pre-Listing Iterator Files Before Execution

Before any node executes, the runner scans every ForEachNode folder to build a deterministic file list. This happens in listIteratorFiles (lines 15-33):

// Runner – resolve files for each iterator
for (const w of workflow.nodes) {
  if (!isIterator(w.type)) continue;
  const dir = (w.data.params?.dir as string | undefined)?.trim();
  if (!dir) { fail('For Each: pick a folder first', 'No folder selected'); return; }
  const files = await listIteratorFiles(dir, iteratorConfig(w).exts);
  if (files.length === 0) { fail(`For Each: no matching files in ${dir}`, 'Empty folder'); return; }
  iteratorFiles.set(w.id, files);
}

Key characteristics of this pre-listing:

• Absolute paths are resolved and stored
• Alphabetical sorting ensures deterministic iteration order
• Extension filtering applies based on the selected mode (via iteratorConfig)
• Early failure if folders are empty or unset

The iteratorFiles.set(w.id, files) call (lines 26-33) populates a global map that persists through the entire workflow run.

Phase 2: Loop Table Construction for ForEachNode Batch Processing

The engine unifies all repeating constructs—both While containers and ForEach iterators—into a single loop table. For each ForEachNode, the runner pushes a LoopInfo entry (lines 63-75) containing:

Property	Purpose
firstIdx	Index of the first body node in execution order
lastIdx	Index of the last body node (used for grouping)
bodyNodeIds	Set of all node IDs in the iteration body
iterations	Total count from files.length

This metadata enables the runner to identify iteration boundaries and know exactly which nodes constitute the repeatable work.

Phase 3: Lock-Step Groups for Parallel Iterators

Multiple ForEachNodes sharing the same downstream body are grouped for lock-step execution. The forEachGroups calculation (lines 82-92) identifies iterators with matching lastIdx values.

When the runner advances such a group, all iterators in the group move to their next file simultaneously. The body executes once per combined iteration rather than producing a cartesian product. This design supports patterns like processing paired image/depth files where both must advance together.

Phase 4: Per-Iteration Execution in executeIteratorNode

Each iteration calls executeIteratorNode (lines 57-78), which:

1. Retrieves the file list from ctx.iteratorFiles.get(node.id)
2. Looks up current progress from the global store
3. Resolves the file path at files[current - 1]
4. Processes the asset based on mode:

// Runner – execute one iteration
async function executeIteratorNode(node, ctx, setRunState) {
  const files = ctx.iteratorFiles.get(node.id) ?? [];
  const current = useWorkflowRunStore.getState().whileProgress[node.id]?.current ?? 1;
  const path = files[current - 1];
  if (!path) throw new Error('For Each: no file for this iteration');

  const kind = iteratorConfig(node);
  if (kind.outputType === 'text') {
    const text = await readTextFile(path);
    ctx.nodeOutputs.set(node.id, { text, outputType: 'text' });
  } else {
    ctx.nodeOutputs.set(node.id, { filePath: path, outputType: kind.outputType });
  }
}

• Text mode: Reads file contents via readTextFile and stores the string
• Image/Mesh mode: Passes the absolute file path for downstream nodes to handle

Results land in ctx.nodeOutputs where dependent nodes can consume them.

Pause, Continue, and Retry Handling at Iteration Boundaries

ForEachNode batch processing includes fine-grained flow control through the _pauseRequested flag. When triggered from the UI:

• The runner checks at handleLoopEnd (lines 91-100) whether to pause before advancing
• Continue: Advances all iterators in the group to their next files
• Retry: Re-executes the current iteration without advancing indices—useful for transient failures

Notably, no forced pause occurs at the final iteration. The loop simply terminates when all files are processed.

Progress updates flow back to the UI through bumpWhileProgress, which updates the whileProgress[node.id] structure that ForEachNode.tsx reads for its current/total display.

Key Implementation Files

File	Role in ForEachNode Batch Processing
src/areas/workflows/nodes/ForEachNode.tsx	UI for folder selection, mode picking, and progress display
src/areas/workflows/workflowRunStore.ts	Core engine: file resolution, loop tables, iteration execution, pause/resume
src/areas/workflows/nodeBehaviors.ts	Helper utilities isIterator and iteratorConfig for node identification

Summary

• Pre-listing (listIteratorFiles): Scans and sorts all matching files before execution begins, ensuring deterministic batch processing
• Loop metadata: LoopInfo entries track iteration boundaries and body node sets for each ForEachNode
• Lock-step groups: Parallel iterators with shared bodies advance together, not independently
• Mode-aware execution: Text files are read into memory; images and meshes pass file paths downstream
• Boundary-aware control: Pause, Continue, and Retry operate at iteration boundaries without interrupting mid-body execution

Frequently Asked Questions

What file extensions does the ForEachNode support for each mode?

The iteratorConfig helper in nodeBehaviors.ts maps modes to extensions. Image mode typically covers common image formats, text mode handles .txt and similar, and mesh mode targets 3D formats—though exact extensions depend on the Modly version. The runner filters the folder scan to only these extensions.

Can multiple ForEachNodes process the same folder with different modes?

Yes. Each ForEachNode maintains its own iteratorFiles entry keyed by node ID. They can point to identical folders while applying different extension filters. If they share downstream body nodes, the lock-step grouping ensures synchronized advancement; otherwise they iterate independently.

Why does the runner pre-list files instead of lazy-loading during iteration?

Pre-listing in workflowRunStore.ts (lines 15-33) provides three advantages: deterministic progress reporting (total is known upfront), early validation of empty folders or permission errors, and stable iteration order through alphabetical sorting. This design prevents mid-workflow surprises from files appearing or disappearing.

How does retry differ from continuing after a pause?

Retry (triggered via _retryRequested) re-executes the current iteration with the same files, useful when a transient error occurred in the body nodes. Continue advances all grouped iterators to their next files and proceeds. The distinction matters for debugging: retry preserves the iteration index while continue increments it.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}