RomM Redis Caching Strategies: Sessions, API Responses, and Job Queues

RomM uses a unified Redis instance configured via REDIS_URL to handle asynchronous session storage, synchronous API response caching, and priority-based background job queues.

The open-source RomM application leverages Redis as a centralized caching layer to manage user sessions, cache expensive API computations, and queue background tasks. According to the rommapp/romm source code, the platform employs distinct Redis clients and data structures optimized for each use case, ensuring high performance and scalability across the retro game management workflow.

Session Storage with Async Redis

RomM persists user sessions using an asynchronous Redis client (async_cache) that provides non-blocking I/O for the FastAPI application layer.

RedisSessionMiddleware Implementation

In backend/handler/auth/middleware/redis_session_middleware.py, the RedisSessionMiddleware class manages session lifecycle using Redis hash data structures. Each session stores data under the key pattern session:<session_id>, enabling distributed stateless authentication across multiple application instances.

The middleware automatically handles session expiration using Redis's native EXPIRE command:

class RedisSessionMiddleware:
    async def load_session(self, request):
        session_id = request.cookies.get("session_id")
        if session_id:
            data = await async_cache.hgetall(f"session:{session_id}")
            return Session(data)
        return Session()

    async def save_session(self, response, session):
        await async_cache.hmset_dict(f"session:{session.id}", session.data)
        await async_cache.expire(f"session:{session.id}", SESSION_TTL)

This implementation ensures that session data remains available for the duration of the SESSION_TTL while automatically cleaning up expired entries without requiring additional database queries.

API Response Caching with Sync Redis

For data-intensive operations like ROM filtering and device authentication, RomM utilizes a synchronous Redis client (sync_cache) defined in backend/handler/redis_handler.py. This client runs with decode_responses=True to handle string serialization automatically.

Cache Implementation Details

The synchronous cache stores frequently accessed data with Time-To-Live (TTL) values to prevent stale data accumulation. Helper functions in backend/utils/cache.py wrap the raw Redis operations, while direct calls use sync_cache.get, sync_cache.setex, and sync_cache.sadd for atomic operations.

The following pattern demonstrates how RomM caches filtered ROM lists with version tracking:

def get_filtered_roms(cache_key, version):
    redis_key = f"filter_values:probe:{cache_key}:v{version}"
    cached = sync_cache.get(redis_key)
    if cached:
        return json.loads(cached)

    result = compute_filters(...)
    sync_cache.setex(redis_key, ROM_FILTERS_CACHE_TTL, json.dumps(result))
    sync_cache.sadd(f"filter_versions:{cache_key}", redis_key)
    return result

This approach reduces database load by serving cached filter results from memory while maintaining cache coherence through Redis sets that track related keys.

Background Job Queues with RQ

RomM implements a priority-based task queue system using the Python RQ (Redis Queue) library, sharing the same Redis server instance but operating on distinct queue objects.

Queue Priorities and Worker Management

In backend/handler/redis_handler.py, RomM defines three distinct queues with different priority levels:

• high_prio_queue: Immediate execution tasks like library refreshes
• default_queue: Standard metadata operations
• low_prio_queue: Background maintenance and bulk updates

Task enqueueing occurs throughout backend/tasks/tasks.py using the following pattern:

# Enqueue a low-priority job

low_prio_queue.enqueue("backend.tasks.tasks.update_metadata", rom_id, retry=False)

# Enqueue a high-priority job

high_prio_queue.enqueue("backend.tasks.tasks.refresh_library", force=True)

Worker processes defined in backend/watcher.py consume jobs from these queues using the RQ Worker class. The get_job_func_name helper function in backend/handler/redis_handler.py safely extracts job function names for logging and error tracking:

def get_job_func_name(job):
    """Safely extract function name from RQ job for monitoring."""
    if hasattr(job, 'func_name'):
        return job.func_name
    return str(job.func)

This architecture ensures that critical user-facing operations (like library scans) process immediately while relegating lengthy metadata fetches to lower-priority queues.

Summary

• Unified Redis Architecture: RomM configures a single Redis instance via REDIS_URL in backend/config/config_manager.py, sharing connection pools across sessions, cache, and queues.
• Async Session Management: The RedisSessionMiddleware uses async_cache (aioredis-compatible) to store session hashes with automatic expiration, eliminating the need for server-side session state.
• Synchronous API Caching: The sync_cache client handles ROM filter results and authentication tokens with TTL-based expiration, implemented in backend/utils/cache.py and backend/handler/redis_handler.py.
• Priority Job Processing: Three RQ queues (high_prio_queue, default_queue, low_prio_queue) manage background tasks with dedicated workers in backend/watcher.py, ensuring responsive UI performance during intensive operations.

Frequently Asked Questions

How does RomM handle session expiration in Redis?

RomM's RedisSessionMiddleware automatically sets expiration timestamps on session keys using Redis's EXPIRE command. When save_session is called, it executes await async_cache.expire(f"session:{session.id}", SESSION_TTL), ensuring that inactive sessions automatically purge from Redis after the configured timeout period without requiring manual cleanup.

What is the difference between async_cache and sync_cache in RomM?

async_cache is an aioredis-compatible client (redis.asyncio.Redis) used exclusively for session management in the async request cycle, while sync_cache is a synchronous redis.Redis client used for API response caching and data retrieval. The separation prevents blocking the event loop during cache operations while allowing synchronous code paths in background tasks to access cached data efficiently.

How are job priorities managed in RomM's Redis queue system?

RomM creates three distinct RQ Queue instances in backend/handler/redis_handler.py: high_prio_queue, default_queue, and low_prio_queue. Tasks enqueue to specific queues based on urgency—library scans use high priority, while metadata updates use low priority. Workers process jobs from these queues according to RQ's priority semantics, ensuring critical operations complete before background maintenance tasks.

Where is the Redis connection configured in RomM?

The Redis connection URL loads from the environment variable REDIS_URL in backend/config/config_manager.py. This configuration initializes the shared Redis client used by async_cache, sync_cache, and the RQ queue system, ensuring all caching layers connect to the same Redis instance for consistency and simplified deployment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}