How Nitter Performs Cache Migration on Startup: Redis Cleanup Explained

Nitter performs cache migration on startup by initializing a Redis connection pool and executing a sentinel-based migration routine that scans for and deletes legacy cache keys exactly once per namespace.

Nitter, the open-source alternative Twitter frontend, uses Redis to cache Twitter data and reduce API load. When the service launches, it runs an automatic Nitter cache migration process that cleans up stale keys from previous deployments. This ensures the cache remains consistent and prevents storage bloat without requiring manual intervention.

Redis Pool Initialization Triggers Migration

The migration process begins inside initRedisPool in src/nitter.nim. This asynchronous procedure creates a connection pool using configuration values from the Config object, then immediately invokes the migration routine for several legacy namespaces.

proc initRedisPool*(cfg: Config) {.async.} =
  try:
    pool = await newRedisPool(cfg.redisConns, cfg.redisMaxConns,
                              host=cfg.redisHost, port=cfg.redisPort,
                              password=cfg.redisPassword)

Immediately after pool creation, the code calls migrate for multiple legacy key patterns:

await migrate("flatty", "*:*")
await migrate("snappyRss", "rss:*")
await migrate("userBuckets", "p:*")
await migrate("profileDates", "p:*")
await migrate("profileStats", "p:*")
await migrate("userType", "p:*")
await migrate("verifiedType", "p:*")

These calls ensure that outdated cache structures are removed before the application begins serving requests.

The Migration Algorithm and Sentinel Pattern

The actual migration logic resides in migrate within src/redis_cache.nim. This procedure implements a sentinel key pattern that guarantees idempotent execution—each migration runs exactly once regardless of how many times the server restarts.

The routine first checks if a sentinel key exists. If the key is missing (redisNil), the migration proceeds to scan for matching keys, delete them via pipelining, and set the sentinel to prevent future runs:

proc migrate*(key, match: string) {.async.} =
  pool.withAcquire(r):
    let hasKey = await r.get(key)
    if hasKey == redisNil:
      let list = await r.scan(newCursor(0), match, 100000)
      r.startPipelining()
      for item in list:
        dawait r.del(item)
      await r.setk(key, "true")
      dawait r.flushPipeline()

The scan command uses a cursor-based approach with a count of 100000 to handle large datasets without blocking the Redis server. The startPipelining and flushPipeline calls batch the delete operations, minimizing round-trip latency. When the sentinel check fails (key exists), the procedure exits immediately without modifying data.

Legacy Cache Namespaces Cleaned on Startup

According to the source code in src/nitter.nim, Nitter specifically targets these legacy namespaces during startup:

• flatty – Matches all keys with *:* pattern
• snappyRss – Matches RSS-related keys with rss:* pattern
• userBuckets – Matches profile buckets with p:* pattern
• profileDates – Matches profile date caches with p:* pattern
• profileStats – Matches profile statistics with p:* pattern
• userType – Matches user type classifications with p:* pattern
• verifiedType – Matches verification status caches with p:* pattern

Each namespace receives its own sentinel key, allowing granular control over which migrations have completed.

Manual Migration for Testing

Developers can trigger migration manually for testing or debugging purposes. The following example initializes the pool, which runs migrations automatically:

import asyncdispatch, redis_cache
let cfg = loadConfig()               # assume a Config loader exists

waitFor initRedisPool(cfg)           # pool creation runs migration automatically

To force a specific namespace to re-migrate (useful when testing schema changes), manually call the migrate procedure:

waitFor migrate("flatty", "*:*")     # forces a fresh scan & delete of all flatty keys

Note that manual execution respects the same sentinel logic—if the sentinel key exists, no deletion occurs unless you manually remove that sentinel from Redis first.

Summary

• Nitter cache migration runs automatically during initRedisPool in src/nitter.nim at application startup.
• The migrate procedure in src/redis_cache.nim uses a sentinel key pattern to ensure each namespace is cleaned exactly once.
• The routine scans for legacy keys using r.scan with a cursor, deletes them via pipelining, and sets a sentinel flag to "true" upon completion.
• Target namespaces include flatty, snappyRss, userBuckets, profileDates, profileStats, userType, and verifiedType.
• The migration is idempotent and safe to run on every startup without risking data loss or excessive Redis load.

Frequently Asked Questions

What triggers cache migration in Nitter?

Cache migration triggers automatically when the application calls initRedisPool during startup. This procedure creates the Redis connection pool and then sequentially invokes the migrate function for seven specific legacy namespaces defined in src/nitter.nim.

How does Nitter prevent migration from running multiple times?

The system uses a sentinel key pattern. Before scanning for legacy keys, the migrate procedure checks if a sentinel key exists in Redis. If the key returns redisNil, the migration executes and then sets the sentinel to "true". On subsequent startups, the sentinel exists, so the routine exits immediately without redundant scanning or deletion.

Which cache namespaces does Nitter migrate on startup?

According to the source code, Nitter migrates seven specific namespaces: flatty (matching *:*), snappyRss (matching rss:*), and five profile-related namespaces (userBuckets, profileDates, profileStats, userType, and verifiedType) all matching the p:* pattern. Each namespace has its own sentinel key for independent tracking.

Can I manually run Nitter cache migration for testing?

Yes. Developers can import redis_cache and call migrate(namespace, pattern) directly using waitFor or asyncdispatch. To force a re-migration, you must first delete the corresponding sentinel key from Redis, as the procedure checks for this key before executing any deletions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}