# How Mole's Scanner Optimizes Disk Traversal for High-Performance File Analysis

> Discover how Mole's scanner optimizes disk traversal with CPU-aware workers, directory semaphores, and fast-path folding for high-performance file analysis and minimal memory.

- Repository: [Tw93/Mole](https://github.com/tw93/Mole)
- Tags: performance
- Published: 2026-03-20

---

**Mole's scanner optimizes disk traversal by combining CPU-aware concurrent worker pools, directory-level semaphores, fast-path directory folding, and heap-based Top-N collection to deliver high-speed filesystem analysis with minimal memory footprint.**

Mole, an open-source disk cleanup tool by tw93/Mole, implements a sophisticated scanning engine designed to handle massive directories efficiently. Understanding how Mole's scanner optimizes disk traversal reveals the engineering techniques that enable rapid analysis of multi-terabyte drives while maintaining responsive UI updates and accurate space calculations.

## Concurrent Worker Pool Architecture

The scanner implements a multi-layered concurrency model that prevents goroutine explosion while maximizing CPU utilization.

### CPU-Aware Goroutine Limits

In [`cmd/analyze/scanner.go`](https://github.com/tw93/Mole/blob/main/cmd/analyze/scanner.go), the `scanPathConcurrent` function dynamically calculates worker limits based on system resources:

```go
numWorkers := max(min(max(runtime.NumCPU()*cpuMultiplier, minWorkers), maxWorkers, len(children)), 1)
sem := make(chan struct{}, numWorkers)

```

This formula ensures the scanner uses at least `minWorkers` but never exceeds `maxWorkers` or the number of available children, preventing both under-utilization and resource exhaustion.

### Directory-Level Semaphores

The scanner employs separate semaphores for different operation types to prevent specific bottlenecks:

- **`dirSem`**: Controls concurrent directory recursion depth
- **`duSem`**: Limits external `du` process concurrency
- **`duQueueSem`**: Manages pending `du` work queue length

These semaphores are declared in [`scanner.go`](https://github.com/tw93/Mole/blob/main/scanner.go) at lines 84-86 and used throughout the file to throttle I/O-heavy operations independently from CPU-bound tasks.

## Fast Path Optimizations and Directory Folding

Mole avoids deep recursion into directories that are known to be large but uninteresting, such as npm caches or `~/Library`.

### Folded Directory Detection

The `shouldFoldDirWithPath` function identifies directories that should be sized with a single `du` call rather than recursive traversal. This logic is invoked in both `scanPathConcurrent` and `calculateDirSizeConcurrent` (lines 98-103 of [`scanner.go`](https://github.com/tw93/Mole/blob/main/scanner.go)).

Folded directories bypass the worker pool entirely, reducing goroutine churn while still providing accurate size measurements via external processes.

### Symlink Deduplication

Symlinks are inspected exactly once and never followed. The scanner checks `child.Type()&fs.ModeSymlink != 0` in both `scanPathConcurrent` and `calculateDirSizeConcurrent` (lines 31-44), ensuring link sizes are counted once while preventing circular references from causing infinite loops.

## Memory-Efficient Collection Strategies

The scanner maintains constant memory usage regardless of directory size by streaming results through heap-based collectors.

### Heap-Based Top-N Tracking

Instead of storing all entries in memory, the scanner uses two min-heaps defined in [`cmd/analyze/heap.go`](https://github.com/tw93/Mole/blob/main/cmd/analyze/heap.go):

- **`entryHeap`**: Tracks the largest directory entries
- **`largeFileHeap`**: Tracks the largest individual files

Background goroutines read from `entryChan` and `largeFileChan`, inserting items into these heaps (lines 97-107 of [`scanner.go`](https://github.com/tw93/Mole/blob/main/scanner.go)). When a heap reaches capacity, the smallest item is evicted, ensuring memory usage remains proportional to N (the desired top count) rather than the total file count.

### Spotlight Integration for macOS

On macOS, the scanner leverages `mdfind` as a fallback acceleration mechanism. The `findLargeFilesWithSpotlight` function (lines 1-5 reference) queries the Spotlight index for large files, which is then merged with the heap-collected results. If Spotlight returns more items than the heap-based scan, the scanner adopts the larger set, ensuring comprehensive coverage without exhaustive traversal.

## Resilience and Accuracy Features

The scanner implements safeguards against pathological filesystem states and provides precise size calculations for modern storage systems.

### Timeout-Protected I/O

All recursive size calculations and external `du` commands execute within a 5-minute timeout context. In `calculateDirSizeFast` (lines 43-46), the scanner creates:

```go
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)

```

This prevents the scanner from hanging on network-mounted drives, permission-denied loops, or corrupted filesystems.

### Accurate Block Allocation

For sparse files or cloud-backed storage (such as Dropbox or iCloud), logical file size differs from actual disk usage. The `getActualFileSize` function (lines 45-55) calculates real consumption by reading `stat.Blocks * 512`, ensuring the scanner reports true disk usage rather than apparent file sizes.

## Core Algorithm Walkthrough

The scanner operates through a structured pipeline:

1. **Entry point** – `scanPathConcurrent` receives the root directory and initializes counters.
2. **Worker allocation** – Runtime CPU detection sizes the semaphore pool dynamically.
3. **Child iteration** – `os.ReadDir` streams immediate children without recursive loading.
4. **Type dispatch** – Symlinks are recorded and skipped; files are sized immediately; directories are classified as folded, special-cased, or recursive.
5. **Concurrent recursion** – Directory workers acquire `dirSem` permits before spawning nested goroutines.
6. **Size calculation** – Folded directories trigger `du` calls (guarded by `duSem`); regular directories use `calculateDirSizeFast` with timeout contexts.
7. **Heap streaming** – Collectors maintain top-N lists via min-heaps, evicting smaller entries as needed.
8. **Spotlight fallback** – macOS systems query `mdfind` to supplement heap results.
9. **Aggregation** – Channels close, heaps drain to sorted slices, and `scanResult` returns comprehensive metadata.

## Practical Implementation Examples

### Running the Scanner via High-Level API

```go
package main

import (
    "fmt"
    "log"
    "os"
    "sync/atomic"
)

func main() {
    var filesScanned, dirsScanned, bytesScanned int64
    currentPath := &atomic.Value{}
    
    result, err := scanPathConcurrent(
        os.Getenv("HOME"),
        &filesScanned, &dirsScanned, &bytesScanned,
        currentPath,
    )
    if err != nil {
        log.Fatalf("scan failed: %v", err)
    }

    fmt.Printf("Total size: %d bytes across %d files\n", 
        result.TotalSize, result.TotalFiles)
    fmt.Println("Top 5 entries:")
    for i, e := range result.Entries[:min(5, len(result.Entries))] {
        fmt.Printf("%d. %s (%d bytes)\n", i+1, e.Path, e.Size)
    }
}

```

*The `scanPathConcurrent` function is defined in* [[`cmd/analyze/scanner.go`](https://github.com/tw93/Mole/blob/main/cmd/analyze/scanner.go)](https://github.com/tw93/Mole/blob/main/cmd/analyze/scanner.go).

### Customizing Worker Limits for Resource Constraints

```go
// Override default CPU multiplier for low-end machines or containers
func init() {
    cpuMultiplier = 1 // Restrict to NumCPU() workers instead of 2x
}

```

*Constants `cpuMultiplier`, `minWorkers`, and `maxWorkers` are declared near the top of* [[`scanner.go`](https://github.com/tw93/Mole/blob/main/scanner.go)](https://github.com/tw93/Mole/blob/main/cmd/analyze/scanner.go).

### Using the Fast Size-Only Path

```go
// For background size calculations without large-file tracking
var files, dirs, bytes int64
size := calculateDirSizeFast("/var/log", &files, &dirs, &bytes, nil)
fmt.Printf("Log directory uses %d bytes (%d files)\n", size, files)

```

*Implementation resides in* [[`scanner.go`](https://github.com/tw93/Mole/blob/main/scanner.go)](https://github.com/tw93/Mole/blob/main/cmd/analyze/scanner.go#L38-L45).

## Key Source Files

| File | Purpose |
|------|---------|
| **[`cmd/analyze/scanner.go`](https://github.com/tw93/Mole/blob/main/cmd/analyze/scanner.go)** | Core scanning engine implementing concurrency controls, directory folding, heap collection, and Spotlight integration. |
| **[`cmd/analyze/heap.go`](https://github.com/tw93/Mole/blob/main/cmd/analyze/heap.go)** | Min-heap definitions for `entryHeap` and `largeFileHeap` used in constant-memory Top-N tracking. |
| **[`cmd/analyze/view.go`](https://github.com/tw93/Mole/blob/main/cmd/analyze/view.go)** | CLI formatting layer that renders `scanResult` for terminal output. |
| **[`cmd/analyze/json.go`](https://github.com/tw93/Mole/blob/main/cmd/analyze/json.go)** | JSON serialization for scan results to support downstream tooling. |
| **[`bin/optimize.sh`](https://github.com/tw93/Mole/blob/main/bin/optimize.sh)** | Wrapper script invoking the scanner with optimized defaults for the `mole optimize` command. |
| **[`cmd/status/metrics_disk.go`](https://github.com/tw93/Mole/blob/main/cmd/status/metrics_disk.go)** | Reuses `getActualFileSize` and `getDirectorySizeFromDu` for real-time disk metrics. |

## Summary

Mole's scanner optimizes disk traversal through several complementary strategies:

- **Adaptive concurrency** using CPU-aware worker pools and separate semaphores for directories and external processes
- **Fast-path folding** that bypasses deep recursion for known heavy directories like npm caches or `~/Library`
- **Constant-memory collection** via min-heaps that track only the Top-N largest files and directories
- **macOS Spotlight integration** to supplement filesystem walks with indexed metadata
- **Timeout-protected I/O** preventing hangs on network drives or corrupted filesystems
- **Accurate block-level sizing** reporting true disk usage rather than logical file sizes for sparse and cloud-backed files

These techniques combine to deliver sub-linear memory growth with near-linear scaling performance across multi-terabyte volumes.

## Frequently Asked Questions

### How does Mole prevent memory exhaustion when scanning directories with millions of files?

Mole uses heap-based Top-N collection instead of storing all entries. Two min-heaps (`entryHeap` and `largeFileHeap`) defined in [`cmd/analyze/heap.go`](https://github.com/tw93/Mole/blob/main/cmd/analyze/heap.go) maintain only the largest N items. When a new item exceeds the smallest heap member, it evicts the smallest element, ensuring memory usage remains constant regardless of total file count.

### Why does Mole use external `du` processes instead of pure Go for size calculation?

External `du` processes provide optimized block-level counting that handles hard links, sparse files, and filesystem-specific attributes more efficiently than standard library calls. The scanner throttles these processes using `duSem` and `duQueueSem` semaphores to prevent process table exhaustion, falling back to `calculateDirSizeFast` when `du` is unavailable or times out.

### How does Mole handle circular symlinks or mount point loops?

The scanner detects symlinks by checking `child.Type()&fs.ModeSymlink != 0` in both `scanPathConcurrent` and `calculateDirSizeConcurrent`. When identified, symlinks are recorded once with their link size but never followed, preventing infinite recursion. The `calculateDirSizeFast` function additionally uses a 5-minute timeout context to escape pathological filesystem states.

### What is the difference between `scanPathConcurrent` and `calculateDirSizeFast`?

`scanPathConcurrent` provides the full analysis pipeline including large-file tracking, heap collection, and metadata gathering, making it suitable for interactive use. `calculateDirSizeFast` offers a lightweight alternative that calculates total size and file counts without maintaining Top-N heaps or tracking individual large files, ideal for background metrics or quick size checks.