How Nitter Performs Cache Migration for Outdated Redis Key Patterns

Nitter automatically migrates outdated Redis key patterns at startup by checking sentinel keys, scanning for legacy patterns with non-blocking SCAN operations, and deleting orphaned keys in batches using Redis pipelining.

Nitter is an alternative Twitter frontend that relies heavily on Redis for caching tweets, user profiles, and RSS feeds. As the codebase evolves, internal cache key naming conventions change between releases, potentially leaving stale keys that waste memory or serve outdated data. The application handles this automatically at boot time through a migration system implemented in src/redis_cache.nim designed to clean up legacy Redis key patterns efficiently.

The Migration Architecture

Initialization via initRedisPool

When Nitter boots, the initRedisPool procedure in src/redis_cache.nim establishes the Redis connection pool using configuration values from src/config.nim. Immediately after pool creation, it invokes the migrate procedure for each known legacy key pattern to ensure the cache remains consistent with the current schema.

Sentinel Key Pattern

The migration system uses a sentinel key to ensure idempotency. Before processing, migrate checks if a specific sentinel key exists using await r.exists(key). If found, the migration skips execution. If absent, the procedure proceeds to scan for and delete legacy keys, then creates the sentinel with value "true" to prevent future runs. This guarantees the migration executes exactly once per deployment, even across server restarts.

Non-Blocking Scan and Pipeline Deletion

The migrate procedure uses Redis's SCAN command (via r.scan) with a cursor starting at 0 and a count of 100000 to enumerate keys matching the old pattern without blocking the server. It then initiates a pipeline with r.startPipelining(), queues DEL commands for each discovered key, sets the sentinel key using await r.setk(key, "true"), and flushes the entire pipeline with await r.flushPipeline(). This asynchronous approach minimizes round trips and keeps startup latency low.

Legacy Key Patterns Migrated

Nitter's initRedisPool (lines 49-55) invokes migrations for several historic key families:

• flatty: Pattern *:* — Removes legacy flatty-encoded objects
• snappyRss: Pattern rss:* — Cleans RSS feeds stored with previous compression schemes
• userBuckets: Pattern p:* — Removes per-user bucket structures
• profileDates: Pattern p:* — Removes cached profile date information
• profileStats: Pattern p:* — Removes profile statistics
• userType: Pattern p:* — Removes cached user-type flags
• verifiedType: Pattern p:* — Removes verified-status flags

Code Implementation Examples

Automatic Migration at Startup

During normal server initialization in src/nitter.nim, the migration triggers automatically through initRedisPool:

# Load configuration from src/config.nim

let cfg = Config.load()

# Initialize pool and run migrations defined in src/redis_cache.nim

await initRedisPool(cfg)

Adding a New Migration

To handle future key pattern changes, add a new migrate call inside initRedisPool before the existing calls:

# In src/redis_cache.nim, append to the initialization sequence:

await migrate("newFeatureCache", "oldfeat:*")

Manual Migration for Testing

For debugging or forced cleanup, call the migrate procedure directly with a custom sentinel:

# Direct invocation to force specific pattern deletion

await migrate("oldCacheMarker", "oldcache:*")

Summary

• Nitter handles Redis schema changes automatically at startup through the initRedisPool procedure in src/redis_cache.nim.
• The sentinel key pattern ensures migrations run only once per deployment, preventing redundant operations across restarts.
• Legacy keys are discovered using non-blocking SCAN with a cursor and count of 100000, then deleted efficiently via Redis pipelining.
• Multiple historic patterns are handled, including *:* for flatty encoding and p:* for various user-related caches like userBuckets and profileStats.
• The entire process is asynchronous, ensuring minimal impact on server startup time even with large Redis datasets.

Frequently Asked Questions

How does Nitter prevent migrations from running on every restart?

The system checks for a specific sentinel key before executing each migration. If the sentinel exists, the procedure returns immediately. Only after successfully deleting all legacy keys and flushing the pipeline does the code execute await r.setk(key, "true") to create the sentinel. This guarantees idempotent execution across server restarts.

What happens if a migration fails halfway through?

Because the sentinel key is only set after the pipeline successfully flushes all DEL commands, a failed or interrupted migration will not create the sentinel. On the next startup, Nitter will detect the missing sentinel and retry the migration from the beginning, ensuring eventual consistency without manual intervention.

Why does Nitter use SCAN instead of KEYS for finding legacy patterns?

The migrate procedure uses SCAN (via r.scan) with a cursor-based approach instead of KEYS because KEYS blocks the Redis server while iterating over the entire keyspace. SCAN provides a non-blocking alternative that iterates incrementally using a cursor starting at 0 and a large count parameter (100000), preventing performance degradation during startup on large Redis instances.

Where are the migration calls configured in the codebase?

All migration invocations are hardcoded in the initRedisPool procedure between lines 49-55 of src/redis_cache.nim. Each call specifies a unique sentinel identifier and the glob pattern to match, such as await migrate("flatty", "*:*") for removing legacy flatty-encoded objects or await migrate("snappyRss", "rss:*") for old RSS feed compression schemes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}