# How RomM Library Scanning Works: A Technical Deep Dive into the Backend Pipeline

> Discover how RomM's library scanning works with this technical deep dive. Learn about its backend pipeline from filesystem changes to frontend updates via Redis-RQ and Socket.IO.

- Repository: [The RomM Project/romm](https://github.com/rommapp/romm)
- Tags: deep-dive
- Published: 2026-07-07

---

**RomM's library scanning is a coordinated backend workflow that watches filesystem changes, enqueues scan jobs via Redis-RQ, processes ROM files with metadata enrichment, and emits real-time progress updates via Socket.IO to the Vue frontend.**

RomM is a self-hosted web application for organizing retro game collections. Its **library scanning** feature automatically discovers ROM files, enriches them with metadata from gaming databases, and synchronizes the SQLAlchemy database with your filesystem. Understanding this pipeline is essential for troubleshooting scan failures and optimizing performance for large collections.

## Entry Point: Triggering Scans via the API

When a user initiates a scan from the Vue frontend, the application calls `POST /api/scan`. This endpoint is defined in [`backend/endpoints/tasks.py`](https://github.com/rommapp/romm/blob/main/backend/endpoints/tasks.py) and immediately forwards the request to the Redis-RQ queue rather than processing synchronously.

```python

# Backend: the endpoint that queues a scan job

# File: backend/endpoints/tasks.py

@router.post('/scan')
async def scan_library(request: Request):
    await request.app.state.rq.enqueue_job('scan_library')
    return JSONResponse({'status': 'queued'})

```

### Scan Handler and Job Coordination

Once the API receives the request, the **scan handler** ([`backend/handler/scan_handler.py`](https://github.com/rommapp/romm/blob/main/backend/handler/scan_handler.py)) creates a new scan job and manages its lifecycle. This handler coordinates between the queue and the actual scanning logic, ensuring that only one intensive scan operation runs at a time while maintaining progress state.

## The Scanning Pipeline: From Filesystem to Database

The core scanning logic resides in `ScanLibraryTask`, a periodic task defined in [`backend/tasks/scheduled/scan_library.py`](https://github.com/rommapp/romm/blob/main/backend/tasks/scheduled/scan_library.py). This class extends `PeriodicTask` and runs automatically on startup as well as when triggered manually.

### Filesystem Discovery with Watcher

Inside the task, RomM utilizes the `Watcher` utility ([`backend/watcher.py`](https://github.com/rommapp/romm/blob/main/backend/watcher.py)) to traverse directories configured in your `.env` or [`config.yml`](https://github.com/rommapp/romm/blob/main/config.yml) file. The watcher detects new, modified, or removed files and yields absolute paths for each ROM file it discovers.

```python

# Core scan task – walks the library and creates ROM entries

# File: backend/tasks/scheduled/scan_library.py

class ScanLibraryTask(PeriodicTask):
    async def run(self) -> None:
        watcher = Watcher(self.settings.library_paths)
        async for rom_path in watcher.iter_files():
            await process_rom(rom_path)   # creates Rom model, fetches metadata, etc.

```

### ROM Processing and Metadata Enrichment

For each discovered file, RomM instantiates a `Rom` ORM object ([`backend/models/rom.py`](https://github.com/rommapp/romm/blob/main/backend/models/rom.py)). The system then invokes **metadata adapters** located in `backend/adapters/` to fetch game titles, descriptions, cover art, and other details from providers like IGDB, TheGamesDB, and OpenVGDB.

### Hashing and Deduplication

Before committing to the database, RomM computes a deterministic hash using the utility in [`backend/utils/rvz_hasher.py`](https://github.com/rommapp/romm/blob/main/backend/utils/rvz_hasher.py). This hash ensures that duplicate ROM files are not created in the database, even if the same game exists in multiple formats or locations.

### Database Commit and Search Indexing

After populating the ROM record, the task commits the SQLAlchemy session to the database. Simultaneously, it updates the **search index** (managed via the Elasticsearch-compatible index created in Alembic migration [`0084_add_roms_search_index.py`](https://github.com/rommapp/romm/blob/main/0084_add_roms_search_index.py)) to ensure new entries appear immediately in the frontend search interface.

## Real-Time Progress Feedback

To keep users informed, RomM emits **Socket.IO** events during the scan process. The endpoint [`backend/endpoints/sockets/scan.py`](https://github.com/rommapp/romm/blob/main/backend/endpoints/sockets/scan.py) broadcasts `scan_progress` events that contain the current status, including counts of processed, skipped, and failed items.

The Vue frontend listens for these events through the Pinia store defined in [`frontend/src/stores/scanning.ts`](https://github.com/rommapp/romm/blob/main/frontend/src/stores/scanning.ts):

```typescript
// Front‑end: trigger a full library scan
import { useApi } from '@/services/api';
import { useScanning } from '@/stores/scanning';

async function startScan() {
  const api = useApi();
  await api.post('/scan');          // calls the backend endpoint
  const scanning = useScanning();
  scanning.start();                // shows progress UI
}

```

Internationalized strings for the scanning interface are stored in [`frontend/src/locales/en_US/scan.json`](https://github.com/rommapp/romm/blob/main/frontend/src/locales/en_US/scan.json) and referenced by the UI components to display localized progress messages.

## Summary

- **Async Queue Architecture**: RomM offloads scanning to Redis-RQ via [`backend/endpoints/tasks.py`](https://github.com/rommapp/romm/blob/main/backend/endpoints/tasks.py), preventing HTTP request timeouts during long operations.
- **Filesystem Monitoring**: The `Watcher` class in [`backend/watcher.py`](https://github.com/rommapp/romm/blob/main/backend/watcher.py) efficiently detects changes in configured library paths without blocking the main thread.
- **Metadata Integration**: Multiple adapter classes in `backend/adapters/` enrich ROM records with cover art and game details from external APIs.
- **Real-Time Updates**: Socket.IO events emitted from [`backend/endpoints/sockets/scan.py`](https://github.com/rommapp/romm/blob/main/backend/endpoints/sockets/scan.py) provide live progress bars in the Vue frontend via [`frontend/src/stores/scanning.ts`](https://github.com/rommapp/romm/blob/main/frontend/src/stores/scanning.ts).
- **Search Integration**: New ROMs are immediately indexed through Alembic migration [`0084_add_roms_search_index.py`](https://github.com/rommapp/romm/blob/main/0084_add_roms_search_index.py), ensuring instant searchability.

## Frequently Asked Questions

### How does RomM handle large libraries without timing out?

RomM uses **Redis-RQ** to queue scan jobs asynchronously. When you trigger a scan via `POST /api/scan`, the endpoint immediately returns a "queued" status and delegates the work to a background worker. This prevents HTTP timeouts and allows the frontend to display real-time progress via Socket.IO events while the backend processes files.

### What metadata sources does RomM use during scanning?

During the scanning pipeline, RomM invokes **metadata adapters** located in `backend/adapters/` to fetch game information. These adapters connect to external providers such as IGDB, TheGamesDB, and OpenVGDB to retrieve titles, descriptions, release dates, and cover art, which are then stored in the `Rom` model defined in [`backend/models/rom.py`](https://github.com/rommapp/romm/blob/main/backend/models/rom.py).

### How does RomM prevent duplicate ROM entries?

RomM computes a deterministic hash for each file using [`backend/utils/rvz_hasher.py`](https://github.com/rommapp/romm/blob/main/backend/utils/rvz_hasher.py) before creating database records. This hash acts as a deduplication key, ensuring that identical ROM files—whether they exist in multiple formats or directory locations—are not imported as separate entries in the SQLAlchemy database.

### Why don't new ROMs appear in search immediately after scanning?

If new ROMs are missing from search results, the **search index** may not have updated properly. The scan task automatically updates the Elasticsearch-compatible index defined in Alembic migration [`0084_add_roms_search_index.py`](https://github.com/rommapp/romm/blob/main/0084_add_roms_search_index.py) after committing ROM records. Ensure your database migrations are current and the search service is running if using external Elasticsearch.