How Mole's Scanner Optimizes Disk Traversal for High-Performance File Analysis

Mole's scanner optimizes disk traversal by combining CPU-aware concurrent worker pools, directory-level semaphores, fast-path directory folding, and heap-based Top-N collection to deliver high-speed filesystem analysis with minimal memory footprint.

Mole, an open-source disk cleanup tool by tw93/Mole, implements a sophisticated scanning engine designed to handle massive directories efficiently. Understanding how Mole's scanner optimizes disk traversal reveals the engineering techniques that enable rapid analysis of multi-terabyte drives while maintaining responsive UI updates and accurate space calculations.

Concurrent Worker Pool Architecture

The scanner implements a multi-layered concurrency model that prevents goroutine explosion while maximizing CPU utilization.

CPU-Aware Goroutine Limits

In cmd/analyze/scanner.go, the scanPathConcurrent function dynamically calculates worker limits based on system resources:

numWorkers := max(min(max(runtime.NumCPU()*cpuMultiplier, minWorkers), maxWorkers, len(children)), 1)
sem := make(chan struct{}, numWorkers)

This formula ensures the scanner uses at least minWorkers but never exceeds maxWorkers or the number of available children, preventing both under-utilization and resource exhaustion.

Directory-Level Semaphores

The scanner employs separate semaphores for different operation types to prevent specific bottlenecks:

• dirSem: Controls concurrent directory recursion depth
• duSem: Limits external du process concurrency
• duQueueSem: Manages pending du work queue length

These semaphores are declared in scanner.go at lines 84-86 and used throughout the file to throttle I/O-heavy operations independently from CPU-bound tasks.

Fast Path Optimizations and Directory Folding

Mole avoids deep recursion into directories that are known to be large but uninteresting, such as npm caches or ~/Library.

Folded Directory Detection

The shouldFoldDirWithPath function identifies directories that should be sized with a single du call rather than recursive traversal. This logic is invoked in both scanPathConcurrent and calculateDirSizeConcurrent (lines 98-103 of scanner.go).

Folded directories bypass the worker pool entirely, reducing goroutine churn while still providing accurate size measurements via external processes.

Symlink Deduplication

Symlinks are inspected exactly once and never followed. The scanner checks child.Type()&fs.ModeSymlink != 0 in both scanPathConcurrent and calculateDirSizeConcurrent (lines 31-44), ensuring link sizes are counted once while preventing circular references from causing infinite loops.

Memory-Efficient Collection Strategies

The scanner maintains constant memory usage regardless of directory size by streaming results through heap-based collectors.

Heap-Based Top-N Tracking

Instead of storing all entries in memory, the scanner uses two min-heaps defined in cmd/analyze/heap.go:

• entryHeap: Tracks the largest directory entries
• largeFileHeap: Tracks the largest individual files

Background goroutines read from entryChan and largeFileChan, inserting items into these heaps (lines 97-107 of scanner.go). When a heap reaches capacity, the smallest item is evicted, ensuring memory usage remains proportional to N (the desired top count) rather than the total file count.

Spotlight Integration for macOS

On macOS, the scanner leverages mdfind as a fallback acceleration mechanism. The findLargeFilesWithSpotlight function (lines 1-5 reference) queries the Spotlight index for large files, which is then merged with the heap-collected results. If Spotlight returns more items than the heap-based scan, the scanner adopts the larger set, ensuring comprehensive coverage without exhaustive traversal.

Resilience and Accuracy Features

The scanner implements safeguards against pathological filesystem states and provides precise size calculations for modern storage systems.

Timeout-Protected I/O

All recursive size calculations and external du commands execute within a 5-minute timeout context. In calculateDirSizeFast (lines 43-46), the scanner creates:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)

This prevents the scanner from hanging on network-mounted drives, permission-denied loops, or corrupted filesystems.

Accurate Block Allocation

For sparse files or cloud-backed storage (such as Dropbox or iCloud), logical file size differs from actual disk usage. The getActualFileSize function (lines 45-55) calculates real consumption by reading stat.Blocks * 512, ensuring the scanner reports true disk usage rather than apparent file sizes.

Core Algorithm Walkthrough

The scanner operates through a structured pipeline:

1. Entry point – scanPathConcurrent receives the root directory and initializes counters.
2. Worker allocation – Runtime CPU detection sizes the semaphore pool dynamically.
3. Child iteration – os.ReadDir streams immediate children without recursive loading.
4. Type dispatch – Symlinks are recorded and skipped; files are sized immediately; directories are classified as folded, special-cased, or recursive.
5. Concurrent recursion – Directory workers acquire dirSem permits before spawning nested goroutines.
6. Size calculation – Folded directories trigger du calls (guarded by duSem); regular directories use calculateDirSizeFast with timeout contexts.
7. Heap streaming – Collectors maintain top-N lists via min-heaps, evicting smaller entries as needed.
8. Spotlight fallback – macOS systems query mdfind to supplement heap results.
9. Aggregation – Channels close, heaps drain to sorted slices, and scanResult returns comprehensive metadata.

Practical Implementation Examples

Running the Scanner via High-Level API

package main

import (
    "fmt"
    "log"
    "os"
    "sync/atomic"
)

func main() {
    var filesScanned, dirsScanned, bytesScanned int64
    currentPath := &atomic.Value{}
    
    result, err := scanPathConcurrent(
        os.Getenv("HOME"),
        &filesScanned, &dirsScanned, &bytesScanned,
        currentPath,
    )
    if err != nil {
        log.Fatalf("scan failed: %v", err)
    }

    fmt.Printf("Total size: %d bytes across %d files\n", 
        result.TotalSize, result.TotalFiles)
    fmt.Println("Top 5 entries:")
    for i, e := range result.Entries[:min(5, len(result.Entries))] {
        fmt.Printf("%d. %s (%d bytes)\n", i+1, e.Path, e.Size)
    }
}

The scanPathConcurrent function is defined in [cmd/analyze/scanner.go](https://github.com/tw93/Mole/blob/main/cmd/analyze/scanner.go).

Customizing Worker Limits for Resource Constraints

// Override default CPU multiplier for low-end machines or containers
func init() {
    cpuMultiplier = 1 // Restrict to NumCPU() workers instead of 2x
}

Constants cpuMultiplier, minWorkers, and maxWorkers are declared near the top of [scanner.go](https://github.com/tw93/Mole/blob/main/cmd/analyze/scanner.go).

Using the Fast Size-Only Path

// For background size calculations without large-file tracking
var files, dirs, bytes int64
size := calculateDirSizeFast("/var/log", &files, &dirs, &bytes, nil)
fmt.Printf("Log directory uses %d bytes (%d files)\n", size, files)

Implementation resides in [scanner.go](https://github.com/tw93/Mole/blob/main/cmd/analyze/scanner.go#L38-L45).

Key Source Files

File	Purpose
cmd/analyze/scanner.go	Core scanning engine implementing concurrency controls, directory folding, heap collection, and Spotlight integration.
cmd/analyze/heap.go	Min-heap definitions for entryHeap and largeFileHeap used in constant-memory Top-N tracking.
cmd/analyze/view.go	CLI formatting layer that renders scanResult for terminal output.
cmd/analyze/json.go	JSON serialization for scan results to support downstream tooling.
bin/optimize.sh	Wrapper script invoking the scanner with optimized defaults for the mole optimize command.
cmd/status/metrics_disk.go	Reuses getActualFileSize and getDirectorySizeFromDu for real-time disk metrics.

Summary

Mole's scanner optimizes disk traversal through several complementary strategies:

• Adaptive concurrency using CPU-aware worker pools and separate semaphores for directories and external processes
• Fast-path folding that bypasses deep recursion for known heavy directories like npm caches or ~/Library
• Constant-memory collection via min-heaps that track only the Top-N largest files and directories
• macOS Spotlight integration to supplement filesystem walks with indexed metadata
• Timeout-protected I/O preventing hangs on network drives or corrupted filesystems
• Accurate block-level sizing reporting true disk usage rather than logical file sizes for sparse and cloud-backed files

These techniques combine to deliver sub-linear memory growth with near-linear scaling performance across multi-terabyte volumes.

Frequently Asked Questions

How does Mole prevent memory exhaustion when scanning directories with millions of files?

Mole uses heap-based Top-N collection instead of storing all entries. Two min-heaps (entryHeap and largeFileHeap) defined in cmd/analyze/heap.go maintain only the largest N items. When a new item exceeds the smallest heap member, it evicts the smallest element, ensuring memory usage remains constant regardless of total file count.

Why does Mole use external du processes instead of pure Go for size calculation?

External du processes provide optimized block-level counting that handles hard links, sparse files, and filesystem-specific attributes more efficiently than standard library calls. The scanner throttles these processes using duSem and duQueueSem semaphores to prevent process table exhaustion, falling back to calculateDirSizeFast when du is unavailable or times out.

How does Mole handle circular symlinks or mount point loops?

The scanner detects symlinks by checking child.Type()&fs.ModeSymlink != 0 in both scanPathConcurrent and calculateDirSizeConcurrent. When identified, symlinks are recorded once with their link size but never followed, preventing infinite recursion. The calculateDirSizeFast function additionally uses a 5-minute timeout context to escape pathological filesystem states.

What is the difference between scanPathConcurrent and calculateDirSizeFast?

scanPathConcurrent provides the full analysis pipeline including large-file tracking, heap collection, and metadata gathering, making it suitable for interactive use. calculateDirSizeFast offers a lightweight alternative that calculates total size and file counts without maintaining Top-N heaps or tracking individual large files, ideal for background metrics or quick size checks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}