Performance Considerations for Scanning Large Media Directories Using the scandirRecursive Helper Function

The scandirRecursive helper in the OpenMage Image Cleaner module accumulates every file path in a single in-memory array, which can exhaust RAM, trigger recursion limits, and exceed PHP execution timeouts when scanning Magento media directories containing tens of thousands of files.

The OpenMage Image Cleaner extension scans the Magento media folder to identify orphaned images no longer referenced in the database. At the heart of this operation lies the scandirRecursive method located in app/code/community/Fballiano/ImageCleaner/Helper/Data.php. While effective for small catalogs, this recursive implementation presents significant performance risks for large-scale deployments.

How the scandirRecursive Function Works

The helper method implements a depth-first traversal of the filesystem:

// app/code/community/Fballiano/ImageCleaner/Helper/Data.php
public function scandirRecursive($dir)
{
    $result = array();
    $blacklist_patterns = $this->getBlacklistedPatterns();
    $root = scandir($dir);
    foreach ($root as $value) {
        if ($value === '.' or $value === '..' or $value === 'cache' or $value === 'watermark' or $value === 'optimized' or $value === '.thumbs') continue;
        if ($this->isBlacklisted("$dir/$value", $blacklist_patterns)) continue;
        if (is_file("$dir/$value")) {
            $result[] = "$dir/$value";
            continue;
        }

        foreach ($this->scandirRecursive("$dir/$value") as $value) {
            $result[] = $value;
        }
    }
    return $result;
}

The execution flow follows four distinct steps:

1. Directory ingestion – PHP’s native scandir loads all entries for the current directory into an array.
2. Noise filtering – Hardcoded exclusions (cache, watermark, optimized, .thumbs) and configurable blacklist patterns skip irrelevant branches.
3. Path accumulation – Every discovered file is appended to the $result array via $result[] = "$dir/$value".
4. Recursion – Subdirectories trigger nested calls, with results merged into the parent’s array.

Performance Bottlenecks in Large Media Directories

When the Magento media/catalog/product tree grows beyond modest sizes, the scandirRecursive implementation exhibits five critical failure modes:

Issue	Technical Explanation
Full in-memory accumulation	The $result array holds every absolute file path simultaneously. A catalog with 100,000 images can generate an array consuming 200–300 MB of RAM.
Deep recursion depth	Each nested directory level adds a stack frame. PHP’s default max_nesting_level (256) can be exceeded by deeply cached image paths or custom folder hierarchies.
Repeated system calls	Every directory triggers a discrete scandir syscall. Over NFS or EFS-mounted storage, network latency multiplies linearly with directory count.
Synchronous blocking	The scan executes within the admin request lifecycle. Long-running scans exceed typical max_execution_time limits (30–60 seconds), causing fatal errors or incomplete data.
Lack of early termination	The algorithm walks the entire tree regardless of whether the caller requires only a specific subset (e.g., a single product’s gallery folder).

Memory and Recursion Risks

The recursive accumulation pattern creates a compound risk profile. As the function traverses deeper into the directory tree, two resource consumption curves rise simultaneously:

1. Call stack growth – Each recursive invocation pushes local variables ($result, $blacklist_patterns, $root) onto the PHP stack. While individual frames are small, hundreds of nested directories can trigger Fatal error: Maximum function nesting level of '256' reached.

2. Heap growth – The $result array in the top-level call must hold references to all strings returned by deeper recursive calls. PHP’s copy-on-write behavior prevents immediate duplication, but the array’s internal hash table still requires continuous reallocation as it grows.

For a Magento installation with 500,000 product images distributed across hash-based subdirectories (e.g., media/catalog/product/a/b/abc123.jpg), the resulting array indices alone can exhaust a 512 MB memory_limit.

Optimized Alternatives for High-Volume Stores

Replacing the array-accumulating recursion with generators or SPL iterators eliminates memory bottlenecks by yielding one file path at a time.

Generator-Based Implementation

This approach uses yield to create a lazy iterator that consumes constant memory regardless of directory size:

// app/code/community/Fballiano/ImageCleaner/Helper/Data.php
public function scandirRecursiveGenerator($dir, $depth = 0, $maxDepth = 10)
{
    if ($depth > $maxDepth) {
        return; // Prevent stack overflow on deep trees
    }

    $blacklist = $this->getBlacklistedPatterns();
    $skipDirs = ['.', '..', 'cache', 'watermark', 'optimized', '.thumbs'];

    foreach (new DirectoryIterator($dir) as $fileInfo) {
        $name = $fileInfo->getFilename();

        if (in_array($name, $skipDirs)) {
            continue;
        }

        if ($this->isBlacklisted($fileInfo->getPathname(), $blacklist)) {
            continue;
        }

        if ($fileInfo->isFile()) {
            yield $fileInfo->getPathname();
        } elseif ($fileInfo->isDir()) {
            yield from $this->scandirRecursiveGenerator($fileInfo->getPathname(), $depth + 1, $maxDepth);
        }
    }
}

Key improvements:

• yield emits one path at a time, keeping memory usage flat.
• $maxDepth parameter prevents runaway recursion on malformed directory structures.
• DirectoryIterator is faster than scandir because it avoids creating intermediate arrays for directory entries.

Chunked Processing for Batch Operations

When the full file list is required for database comparison, process the generator in chunks to bound memory usage:

$helper   = Mage::helper('fballiano_imagecleaner');
$mediaDir = Mage::getBaseDir('media') . '/catalog/product';
$chunkSize = 5000;
$buffer = [];

foreach ($helper->scandirRecursiveGenerator($mediaDir) as $filePath) {
    $buffer[] = $filePath;

    if (count($buffer) >= $chunkSize) {
        $this->_processFileChunk($buffer);
        $buffer = []; // Explicitly free memory
    }
}

// Final flush
if (!empty($buffer)) {
    $this->_processFileChunk($buffer);
}

This pattern ensures that PHP’s peak memory usage never exceeds the $chunkSize multiplied by the average path string length, typically keeping consumption under 50 MB even for million-file catalogs.

SPL Iterator Implementation

For developers preferring object-oriented iteration, RecursiveDirectoryIterator provides native depth control and symlink handling:

public function scandirRecursiveIter($dir, $maxDepth = 10)
{
    $iterator = new RecursiveIteratorIterator(
        new RecursiveDirectoryIterator($dir, FilesystemIterator::SKIP_DOTS | FilesystemIterator::CURRENT_AS_FILEINFO),
        RecursiveIteratorIterator::SELF_FIRST
    );
    $iterator->setMaxDepth($maxDepth);

    $blacklist = $this->getBlacklistedPatterns();
    $skipDirs = ['cache', 'watermark', 'optimized', '.thumbs'];

    foreach ($iterator as $fileInfo) {
        $name = $fileInfo->getFilename();

        // Skip system directories
        if (in_array($name, $skipDirs)) {
            $iterator->skipChildren();
            continue;
        }

        if ($this->isBlacklisted($fileInfo->getPathname(), $blacklist)) {
            if ($fileInfo->isDir()) {
                $iterator->skipChildren();
            }
            continue;
        }

        if ($fileInfo->isFile()) {
            yield $fileInfo->getPathname();
        }
    }
}

Advantages of this approach:

• setMaxDepth enforces limits without manual tracking.
• skipChildren prunes entire branches (e.g., the cache folder) without descending into them, reducing system calls.
• FilesystemIterator::SKIP_DOTS eliminates the need for manual . and .. checks.

Integration Guide

To deploy these optimizations in app/code/community/Fballiano/ImageCleaner/controllers/Adminhtml/FbimagecleanerController.php, replace the existing helper invocation:

// Original memory-intensive call
$fs_images = Mage::helper('fballiano_imagecleaner')->scandirRecursive($media_dir);

With the generator-based implementation:

// Memory-efficient generator usage
$fs_images = iterator_to_array(
    Mage::helper('fballiano_imagecleaner')->scandirRecursiveGenerator($media_dir)
);

For extremely large catalogs, implement chunked processing within the controller action to avoid loading the entire filesystem array into memory before database comparison.

Summary

• Memory exhaustion is the primary risk when using the default scandirRecursive implementation on large Magento media directories, as it accumulates all file paths in a single array.
• Recursion depth limits pose a secondary threat on catalogs with deeply nested hash-based directory structures.
• Generator-based alternatives using yield or RecursiveDirectoryIterator provide constant memory usage regardless of catalog size.
• Chunked processing allows the module to handle million-file catalogs by bounding memory usage to a fixed batch size.
• Depth limits and early pruning via skipChildren or maxDepth parameters prevent unnecessary traversal of cache and system directories.

Frequently Asked Questions

What causes PHP memory exhaustion when scanning large media directories?

The default scandirRecursive implementation in app/code/community/Fballiano/ImageCleaner/Helper/Data.php appends every discovered file path to a $result array. On Magento installations with 100,000+ product images, this array can consume hundreds of megabytes of RAM, triggering memory_limit fatal errors before the scan completes.

How does the blacklist feature improve scanning performance?

The blacklist mechanism defined in etc/system.xml and evaluated via isBlacklisted() allows administrators to exclude specific directories (such as import folders or legacy archives) from traversal. By pruning entire branches early, the scanner avoids descending into irrelevant sub-trees, reducing both system call overhead and memory accumulation.

Can the Image Cleaner module operate efficiently on NFS or cloud-mounted media storage?

Network-mounted storage amplifies the performance cost of the default recursive scanner because each scandir call incurs network latency. For NFS, EFS, or S3-backed media directories, use the RecursiveDirectoryIterator implementation with skipChildren to minimize directory listings, or schedule scans during off-peak hours using Magento cron jobs to avoid request timeouts.

What is the maximum safe directory depth for the recursive scanner?

The default implementation has no depth guard and relies on PHP’s max_nesting_level (typically 256). Magento’s standard hash-based directory structure usually reaches 3–4 levels, but custom imports may create deeper nests. Use the generator-based scandirRecursiveGenerator method with an explicit $maxDepth parameter (recommended: 10) to prevent stack overflow errors on deeply nested trees.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}