# Performance Considerations for Scanning Large Media Directories Using the scandirRecursive Helper Function

> Optimize large media directory scanning with scandirRecursive. Avoid RAM exhaustion, recursion limits, and timeouts in OpenMage Image Cleaner. Learn performance best practices.

- Repository: [Fabrizio Balliano/openmage-image-cleaner](https://github.com/fballiano/openmage-image-cleaner)
- Tags: performance
- Published: 2026-03-01

---

**The `scandirRecursive` helper in the OpenMage Image Cleaner module accumulates every file path in a single in-memory array, which can exhaust RAM, trigger recursion limits, and exceed PHP execution timeouts when scanning Magento media directories containing tens of thousands of files.**

The OpenMage Image Cleaner extension scans the Magento `media` folder to identify orphaned images no longer referenced in the database. At the heart of this operation lies the `scandirRecursive` method located in [`app/code/community/Fballiano/ImageCleaner/Helper/Data.php`](https://github.com/fballiano/openmage-image-cleaner/blob/main/app/code/community/Fballiano/ImageCleaner/Helper/Data.php). While effective for small catalogs, this recursive implementation presents significant performance risks for large-scale deployments.

## How the scandirRecursive Function Works

The helper method implements a depth-first traversal of the filesystem:

```php
// app/code/community/Fballiano/ImageCleaner/Helper/Data.php
public function scandirRecursive($dir)
{
    $result = array();
    $blacklist_patterns = $this->getBlacklistedPatterns();
    $root = scandir($dir);
    foreach ($root as $value) {
        if ($value === '.' or $value === '..' or $value === 'cache' or $value === 'watermark' or $value === 'optimized' or $value === '.thumbs') continue;
        if ($this->isBlacklisted("$dir/$value", $blacklist_patterns)) continue;
        if (is_file("$dir/$value")) {
            $result[] = "$dir/$value";
            continue;
        }

        foreach ($this->scandirRecursive("$dir/$value") as $value) {
            $result[] = $value;
        }
    }
    return $result;
}

```

The execution flow follows four distinct steps:

1. **Directory ingestion** – PHP’s native `scandir` loads all entries for the current directory into an array.
2. **Noise filtering** – Hardcoded exclusions (`cache`, `watermark`, `optimized`, `.thumbs`) and configurable blacklist patterns skip irrelevant branches.
3. **Path accumulation** – Every discovered file is appended to the `$result` array via `$result[] = "$dir/$value"`.
4. **Recursion** – Subdirectories trigger nested calls, with results merged into the parent’s array.

## Performance Bottlenecks in Large Media Directories

When the Magento `media/catalog/product` tree grows beyond modest sizes, the `scandirRecursive` implementation exhibits five critical failure modes:

| Issue | Technical Explanation |
|-------|----------------------|
| **Full in-memory accumulation** | The `$result` array holds every absolute file path simultaneously. A catalog with 100,000 images can generate an array consuming 200–300 MB of RAM. |
| **Deep recursion depth** | Each nested directory level adds a stack frame. PHP’s default `max_nesting_level` (256) can be exceeded by deeply cached image paths or custom folder hierarchies. |
| **Repeated system calls** | Every directory triggers a discrete `scandir` syscall. Over NFS or EFS-mounted storage, network latency multiplies linearly with directory count. |
| **Synchronous blocking** | The scan executes within the admin request lifecycle. Long-running scans exceed typical `max_execution_time` limits (30–60 seconds), causing fatal errors or incomplete data. |
| **Lack of early termination** | The algorithm walks the entire tree regardless of whether the caller requires only a specific subset (e.g., a single product’s gallery folder). |

## Memory and Recursion Risks

The **recursive accumulation pattern** creates a compound risk profile. As the function traverses deeper into the directory tree, two resource consumption curves rise simultaneously:

1. **Call stack growth** – Each recursive invocation pushes local variables (`$result`, `$blacklist_patterns`, `$root`) onto the PHP stack. While individual frames are small, hundreds of nested directories can trigger `Fatal error: Maximum function nesting level of '256' reached`.

2. **Heap growth** – The `$result` array in the top-level call must hold references to all strings returned by deeper recursive calls. PHP’s copy-on-write behavior prevents immediate duplication, but the array’s internal hash table still requires continuous reallocation as it grows.

For a Magento installation with 500,000 product images distributed across hash-based subdirectories (e.g., `media/catalog/product/a/b/abc123.jpg`), the resulting array indices alone can exhaust a 512 MB `memory_limit`.

## Optimized Alternatives for High-Volume Stores

Replacing the array-accumulating recursion with **generators** or **SPL iterators** eliminates memory bottlenecks by yielding one file path at a time.

### Generator-Based Implementation

This approach uses `yield` to create a lazy iterator that consumes constant memory regardless of directory size:

```php
// app/code/community/Fballiano/ImageCleaner/Helper/Data.php
public function scandirRecursiveGenerator($dir, $depth = 0, $maxDepth = 10)
{
    if ($depth > $maxDepth) {
        return; // Prevent stack overflow on deep trees
    }

    $blacklist = $this->getBlacklistedPatterns();
    $skipDirs = ['.', '..', 'cache', 'watermark', 'optimized', '.thumbs'];

    foreach (new DirectoryIterator($dir) as $fileInfo) {
        $name = $fileInfo->getFilename();

        if (in_array($name, $skipDirs)) {
            continue;
        }

        if ($this->isBlacklisted($fileInfo->getPathname(), $blacklist)) {
            continue;
        }

        if ($fileInfo->isFile()) {
            yield $fileInfo->getPathname();
        } elseif ($fileInfo->isDir()) {
            yield from $this->scandirRecursiveGenerator($fileInfo->getPathname(), $depth + 1, $maxDepth);
        }
    }
}

```

**Key improvements:**
- **`yield`** emits one path at a time, keeping memory usage flat.
- **`$maxDepth`** parameter prevents runaway recursion on malformed directory structures.
- **`DirectoryIterator`** is faster than `scandir` because it avoids creating intermediate arrays for directory entries.

### Chunked Processing for Batch Operations

When the full file list is required for database comparison, process the generator in chunks to bound memory usage:

```php
$helper   = Mage::helper('fballiano_imagecleaner');
$mediaDir = Mage::getBaseDir('media') . '/catalog/product';
$chunkSize = 5000;
$buffer = [];

foreach ($helper->scandirRecursiveGenerator($mediaDir) as $filePath) {
    $buffer[] = $filePath;

    if (count($buffer) >= $chunkSize) {
        $this->_processFileChunk($buffer);
        $buffer = []; // Explicitly free memory
    }
}

// Final flush
if (!empty($buffer)) {
    $this->_processFileChunk($buffer);
}

```

This pattern ensures that PHP’s peak memory usage never exceeds the `$chunkSize` multiplied by the average path string length, typically keeping consumption under 50 MB even for million-file catalogs.

### SPL Iterator Implementation

For developers preferring object-oriented iteration, `RecursiveDirectoryIterator` provides native depth control and symlink handling:

```php
public function scandirRecursiveIter($dir, $maxDepth = 10)
{
    $iterator = new RecursiveIteratorIterator(
        new RecursiveDirectoryIterator($dir, FilesystemIterator::SKIP_DOTS | FilesystemIterator::CURRENT_AS_FILEINFO),
        RecursiveIteratorIterator::SELF_FIRST
    );
    $iterator->setMaxDepth($maxDepth);

    $blacklist = $this->getBlacklistedPatterns();
    $skipDirs = ['cache', 'watermark', 'optimized', '.thumbs'];

    foreach ($iterator as $fileInfo) {
        $name = $fileInfo->getFilename();

        // Skip system directories
        if (in_array($name, $skipDirs)) {
            $iterator->skipChildren();
            continue;
        }

        if ($this->isBlacklisted($fileInfo->getPathname(), $blacklist)) {
            if ($fileInfo->isDir()) {
                $iterator->skipChildren();
            }
            continue;
        }

        if ($fileInfo->isFile()) {
            yield $fileInfo->getPathname();
        }
    }
}

```

**Advantages of this approach:**
- **`setMaxDepth`** enforces limits without manual tracking.
- **`skipChildren`** prunes entire branches (e.g., the `cache` folder) without descending into them, reducing system calls.
- **`FilesystemIterator::SKIP_DOTS`** eliminates the need for manual `.` and `..` checks.

## Integration Guide

To deploy these optimizations in [`app/code/community/Fballiano/ImageCleaner/controllers/Adminhtml/FbimagecleanerController.php`](https://github.com/fballiano/openmage-image-cleaner/blob/main/app/code/community/Fballiano/ImageCleaner/controllers/Adminhtml/FbimagecleanerController.php), replace the existing helper invocation:

```php
// Original memory-intensive call
$fs_images = Mage::helper('fballiano_imagecleaner')->scandirRecursive($media_dir);

```

With the generator-based implementation:

```php
// Memory-efficient generator usage
$fs_images = iterator_to_array(
    Mage::helper('fballiano_imagecleaner')->scandirRecursiveGenerator($media_dir)
);

```

For extremely large catalogs, implement chunked processing within the controller action to avoid loading the entire filesystem array into memory before database comparison.

## Summary

- **Memory exhaustion** is the primary risk when using the default `scandirRecursive` implementation on large Magento media directories, as it accumulates all file paths in a single array.
- **Recursion depth limits** pose a secondary threat on catalogs with deeply nested hash-based directory structures.
- **Generator-based alternatives** using `yield` or `RecursiveDirectoryIterator` provide constant memory usage regardless of catalog size.
- **Chunked processing** allows the module to handle million-file catalogs by bounding memory usage to a fixed batch size.
- **Depth limits and early pruning** via `skipChildren` or `maxDepth` parameters prevent unnecessary traversal of cache and system directories.

## Frequently Asked Questions

### What causes PHP memory exhaustion when scanning large media directories?

The default `scandirRecursive` implementation in [`app/code/community/Fballiano/ImageCleaner/Helper/Data.php`](https://github.com/fballiano/openmage-image-cleaner/blob/main/app/code/community/Fballiano/ImageCleaner/Helper/Data.php) appends every discovered file path to a `$result` array. On Magento installations with 100,000+ product images, this array can consume hundreds of megabytes of RAM, triggering `memory_limit` fatal errors before the scan completes.

### How does the blacklist feature improve scanning performance?

The blacklist mechanism defined in [`etc/system.xml`](https://github.com/fballiano/openmage-image-cleaner/blob/main/etc/system.xml) and evaluated via `isBlacklisted()` allows administrators to exclude specific directories (such as import folders or legacy archives) from traversal. By pruning entire branches early, the scanner avoids descending into irrelevant sub-trees, reducing both system call overhead and memory accumulation.

### Can the Image Cleaner module operate efficiently on NFS or cloud-mounted media storage?

Network-mounted storage amplifies the performance cost of the default recursive scanner because each `scandir` call incurs network latency. For NFS, EFS, or S3-backed media directories, use the `RecursiveDirectoryIterator` implementation with `skipChildren` to minimize directory listings, or schedule scans during off-peak hours using Magento cron jobs to avoid request timeouts.

### What is the maximum safe directory depth for the recursive scanner?

The default implementation has no depth guard and relies on PHP’s `max_nesting_level` (typically 256). Magento’s standard hash-based directory structure usually reaches 3–4 levels, but custom imports may create deeper nests. Use the generator-based `scandirRecursiveGenerator` method with an explicit `$maxDepth` parameter (recommended: 10) to prevent stack overflow errors on deeply nested trees.