# How the Modly ForEachNode Handles Batch Processing of Assets: A Deep Dive

> Discover how Modly's ForEachNode efficiently handles batch processing. Learn about pre-scanning, file lists, and optional iterator grouping for seamless asset management.

- Repository: [lightningpixel/modly](https://github.com/lightningpixel/modly)
- Tags: deep-dive
- Published: 2026-08-20

---

**The Modly ForEachNode handles batch processing by pre-scanning folders to build an alphabetically-sorted file list, treating each file as a loop iteration, and optionally grouping multiple iterators that share downstream body nodes for lock-step execution.**

The **ForEachNode** is the core iterator that drives batch processing in Modly's workflow engine. If you're working with collections of images, text files, or 3D meshes, understanding how this node manages iteration, progress tracking, and pause/resume functionality is essential for building reliable automation pipelines. This article examines the complete implementation across the UI layer and runtime engine in the `lightningpixel/modly` repository.

## Architecture Overview: UI Component and Runtime Engine

The ForEachNode splits responsibilities between a React-based UI component and the core workflow execution store. This separation keeps the interface responsive while the engine handles file I/O and state management.

### ForEachNode.tsx: Folder Selection and Progress Display

The UI component lives at [`src/areas/workflows/nodes/ForEachNode.tsx`](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/nodes/ForEachNode.tsx). Users interact with two primary controls:

- **Folder picker** (lines 41-61): Opens a native directory dialog via `window.electron.fs.selectDirectory`
- **Asset mode dropdown** (lines 62-87): Selects between `image`, `text`, or `mesh` processing

While a workflow runs, the UI locks these controls and displays iteration progress through a badge showing `current/total` (lines 102-135). Pause, Continue, and Retry buttons appear when the user requests a break at the next iteration boundary.

```tsx
// UI – pick folder and show iteration progress
const pickDir = async () => {
  const picked = await window.electron.fs.selectDirectory(dir || undefined);
  if (picked) updateNodeData(id, { params: { ...data.params, dir: picked } });
};

return (
  <BaseNode …>
    <select … value={mode} onChange={e => updateNodeData(id, { params: { …data.params, mode: e.target.value as Mode, dir: undefined } })}>
      <option value="image">Image</option>
      <option value="text">Text</option>
      <option value="mesh">Mesh</option>
    </select>
    <button onClick={pickDir}>…{dirLabel}</button>
    {progress && <span>{progress.current}/{progress.total}</span>}
  </BaseNode>
);

```

### workflowRunStore.ts: Core Batch Processing Engine

The runtime logic in [`src/areas/workflows/workflowRunStore.ts`](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/workflowRunStore.ts) handles four critical phases: pre-listing files, building loop metadata, executing iterations, and managing flow control.

## Phase 1: Pre-Listing Iterator Files Before Execution

Before any node executes, the runner scans every ForEachNode folder to build a deterministic file list. This happens in `listIteratorFiles` (lines 15-33):

```ts
// Runner – resolve files for each iterator
for (const w of workflow.nodes) {
  if (!isIterator(w.type)) continue;
  const dir = (w.data.params?.dir as string | undefined)?.trim();
  if (!dir) { fail('For Each: pick a folder first', 'No folder selected'); return; }
  const files = await listIteratorFiles(dir, iteratorConfig(w).exts);
  if (files.length === 0) { fail(`For Each: no matching files in ${dir}`, 'Empty folder'); return; }
  iteratorFiles.set(w.id, files);
}

```

Key characteristics of this pre-listing:

- **Absolute paths** are resolved and stored
- **Alphabetical sorting** ensures deterministic iteration order
- **Extension filtering** applies based on the selected mode (via `iteratorConfig`)
- **Early failure** if folders are empty or unset

The `iteratorFiles.set(w.id, files)` call (lines 26-33) populates a global map that persists through the entire workflow run.

## Phase 2: Loop Table Construction for ForEachNode Batch Processing

The engine unifies all repeating constructs—both `While` containers and `ForEach` iterators—into a single **loop table**. For each ForEachNode, the runner pushes a `LoopInfo` entry (lines 63-75) containing:

| Property | Purpose |
|----------|---------|
| `firstIdx` | Index of the first body node in execution order |
| `lastIdx` | Index of the last body node (used for grouping) |
| `bodyNodeIds` | Set of all node IDs in the iteration body |
| `iterations` | Total count from `files.length` |

This metadata enables the runner to identify iteration boundaries and know exactly which nodes constitute the repeatable work.

## Phase 3: Lock-Step Groups for Parallel Iterators

Multiple ForEachNodes sharing the same downstream body are grouped for **lock-step execution**. The `forEachGroups` calculation (lines 82-92) identifies iterators with matching `lastIdx` values.

When the runner advances such a group, **all** iterators in the group move to their next file simultaneously. The body executes once per combined iteration rather than producing a cartesian product. This design supports patterns like processing paired image/depth files where both must advance together.

## Phase 4: Per-Iteration Execution in executeIteratorNode

Each iteration calls `executeIteratorNode` (lines 57-78), which:

1. Retrieves the file list from `ctx.iteratorFiles.get(node.id)`
2. Looks up `current` progress from the global store
3. Resolves the file path at `files[current - 1]`
4. Processes the asset based on mode:

```ts
// Runner – execute one iteration
async function executeIteratorNode(node, ctx, setRunState) {
  const files = ctx.iteratorFiles.get(node.id) ?? [];
  const current = useWorkflowRunStore.getState().whileProgress[node.id]?.current ?? 1;
  const path = files[current - 1];
  if (!path) throw new Error('For Each: no file for this iteration');

  const kind = iteratorConfig(node);
  if (kind.outputType === 'text') {
    const text = await readTextFile(path);
    ctx.nodeOutputs.set(node.id, { text, outputType: 'text' });
  } else {
    ctx.nodeOutputs.set(node.id, { filePath: path, outputType: kind.outputType });
  }
}

```

- **Text mode**: Reads file contents via `readTextFile` and stores the string
- **Image/Mesh mode**: Passes the absolute file path for downstream nodes to handle

Results land in `ctx.nodeOutputs` where dependent nodes can consume them.

## Pause, Continue, and Retry Handling at Iteration Boundaries

ForEachNode batch processing includes fine-grained flow control through the `_pauseRequested` flag. When triggered from the UI:

- The runner checks at `handleLoopEnd` (lines 91-100) whether to pause before advancing
- **Continue**: Advances all iterators in the group to their next files
- **Retry**: Re-executes the current iteration without advancing indices—useful for transient failures

Notably, **no forced pause occurs at the final iteration**. The loop simply terminates when all files are processed.

Progress updates flow back to the UI through `bumpWhileProgress`, which updates the `whileProgress[node.id]` structure that [`ForEachNode.tsx`](https://github.com/lightningpixel/modly/blob/main/ForEachNode.tsx) reads for its `current/total` display.

## Key Implementation Files

| File | Role in ForEachNode Batch Processing |
|------|--------------------------------------|
| [`src/areas/workflows/nodes/ForEachNode.tsx`](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/nodes/ForEachNode.tsx) | UI for folder selection, mode picking, and progress display |
| [`src/areas/workflows/workflowRunStore.ts`](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/workflowRunStore.ts) | Core engine: file resolution, loop tables, iteration execution, pause/resume |
| [`src/areas/workflows/nodeBehaviors.ts`](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/nodeBehaviors.ts) | Helper utilities `isIterator` and `iteratorConfig` for node identification |

## Summary

- **Pre-listing** (`listIteratorFiles`): Scans and sorts all matching files before execution begins, ensuring deterministic batch processing
- **Loop metadata**: `LoopInfo` entries track iteration boundaries and body node sets for each ForEachNode
- **Lock-step groups**: Parallel iterators with shared bodies advance together, not independently
- **Mode-aware execution**: Text files are read into memory; images and meshes pass file paths downstream
- **Boundary-aware control**: Pause, Continue, and Retry operate at iteration boundaries without interrupting mid-body execution

## Frequently Asked Questions

### What file extensions does the ForEachNode support for each mode?

The `iteratorConfig` helper in [`nodeBehaviors.ts`](https://github.com/lightningpixel/modly/blob/main/nodeBehaviors.ts) maps modes to extensions. Image mode typically covers common image formats, text mode handles `.txt` and similar, and mesh mode targets 3D formats—though exact extensions depend on the Modly version. The runner filters the folder scan to only these extensions.

### Can multiple ForEachNodes process the same folder with different modes?

Yes. Each ForEachNode maintains its own `iteratorFiles` entry keyed by node ID. They can point to identical folders while applying different extension filters. If they share downstream body nodes, the lock-step grouping ensures synchronized advancement; otherwise they iterate independently.

### Why does the runner pre-list files instead of lazy-loading during iteration?

Pre-listing in [`workflowRunStore.ts`](https://github.com/lightningpixel/modly/blob/main/workflowRunStore.ts) (lines 15-33) provides three advantages: deterministic progress reporting (`total` is known upfront), early validation of empty folders or permission errors, and stable iteration order through alphabetical sorting. This design prevents mid-workflow surprises from files appearing or disappearing.

### How does retry differ from continuing after a pause?

**Retry** (triggered via `_retryRequested`) re-executes the current iteration with the same files, useful when a transient error occurred in the body nodes. **Continue** advances all grouped iterators to their next files and proceeds. The distinction matters for debugging: retry preserves the iteration index while continue increments it.