# How Nitter Performs Cache Migration on Startup: Redis Cleanup Explained

> Learn how Nitter performs cache migration on startup. Discover the Redis cleanup routine that scans and deletes legacy cache keys once per namespace for efficient operation.

- Repository: [Zed/nitter](https://github.com/zedeus/nitter)
- Tags: internals
- Published: 2026-09-04

---

**Nitter performs cache migration on startup by initializing a Redis connection pool and executing a sentinel-based migration routine that scans for and deletes legacy cache keys exactly once per namespace.**

Nitter, the open-source alternative Twitter frontend, uses Redis to cache Twitter data and reduce API load. When the service launches, it runs an automatic **Nitter cache migration** process that cleans up stale keys from previous deployments. This ensures the cache remains consistent and prevents storage bloat without requiring manual intervention.

## Redis Pool Initialization Triggers Migration

The migration process begins inside **`initRedisPool`** in `src/nitter.nim`. This asynchronous procedure creates a connection pool using configuration values from the `Config` object, then immediately invokes the migration routine for several legacy namespaces.

```nim
proc initRedisPool*(cfg: Config) {.async.} =
  try:
    pool = await newRedisPool(cfg.redisConns, cfg.redisMaxConns,
                              host=cfg.redisHost, port=cfg.redisPort,
                              password=cfg.redisPassword)

```

Immediately after pool creation, the code calls **`migrate`** for multiple legacy key patterns:

```nim
await migrate("flatty", "*:*")
await migrate("snappyRss", "rss:*")
await migrate("userBuckets", "p:*")
await migrate("profileDates", "p:*")
await migrate("profileStats", "p:*")
await migrate("userType", "p:*")
await migrate("verifiedType", "p:*")

```

These calls ensure that outdated cache structures are removed before the application begins serving requests.

## The Migration Algorithm and Sentinel Pattern

The actual migration logic resides in **`migrate`** within `src/redis_cache.nim`. This procedure implements a **sentinel key pattern** that guarantees idempotent execution—each migration runs exactly once regardless of how many times the server restarts.

The routine first checks if a sentinel key exists. If the key is missing (`redisNil`), the migration proceeds to scan for matching keys, delete them via pipelining, and set the sentinel to prevent future runs:

```nim
proc migrate*(key, match: string) {.async.} =
  pool.withAcquire(r):
    let hasKey = await r.get(key)
    if hasKey == redisNil:
      let list = await r.scan(newCursor(0), match, 100000)
      r.startPipelining()
      for item in list:
        dawait r.del(item)
      await r.setk(key, "true")
      dawait r.flushPipeline()

```

The **`scan`** command uses a cursor-based approach with a count of `100000` to handle large datasets without blocking the Redis server. The **`startPipelining`** and **`flushPipeline`** calls batch the delete operations, minimizing round-trip latency. When the sentinel check fails (key exists), the procedure exits immediately without modifying data.

## Legacy Cache Namespaces Cleaned on Startup

According to the source code in `src/nitter.nim`, Nitter specifically targets these legacy namespaces during startup:

- **`flatty`** – Matches all keys with `*:*` pattern
- **`snappyRss`** – Matches RSS-related keys with `rss:*` pattern
- **`userBuckets`** – Matches profile buckets with `p:*` pattern
- **`profileDates`** – Matches profile date caches with `p:*` pattern
- **`profileStats`** – Matches profile statistics with `p:*` pattern
- **`userType`** – Matches user type classifications with `p:*` pattern
- **`verifiedType`** – Matches verification status caches with `p:*` pattern

Each namespace receives its own sentinel key, allowing granular control over which migrations have completed.

## Manual Migration for Testing

Developers can trigger migration manually for testing or debugging purposes. The following example initializes the pool, which runs migrations automatically:

```nim
import asyncdispatch, redis_cache
let cfg = loadConfig()               # assume a Config loader exists

waitFor initRedisPool(cfg)           # pool creation runs migration automatically

```

To force a specific namespace to re-migrate (useful when testing schema changes), manually call the migrate procedure:

```nim
waitFor migrate("flatty", "*:*")     # forces a fresh scan & delete of all flatty keys

```

Note that manual execution respects the same sentinel logic—if the sentinel key exists, no deletion occurs unless you manually remove that sentinel from Redis first.

## Summary

- **Nitter cache migration** runs automatically during `initRedisPool` in `src/nitter.nim` at application startup.
- The **`migrate`** procedure in `src/redis_cache.nim` uses a sentinel key pattern to ensure each namespace is cleaned exactly once.
- The routine scans for legacy keys using **`r.scan`** with a cursor, deletes them via pipelining, and sets a sentinel flag to `"true"` upon completion.
- Target namespaces include `flatty`, `snappyRss`, `userBuckets`, `profileDates`, `profileStats`, `userType`, and `verifiedType`.
- The migration is idempotent and safe to run on every startup without risking data loss or excessive Redis load.

## Frequently Asked Questions

### What triggers cache migration in Nitter?

Cache migration triggers automatically when the application calls **`initRedisPool`** during startup. This procedure creates the Redis connection pool and then sequentially invokes the **`migrate`** function for seven specific legacy namespaces defined in `src/nitter.nim`.

### How does Nitter prevent migration from running multiple times?

The system uses a **sentinel key pattern**. Before scanning for legacy keys, the **`migrate`** procedure checks if a sentinel key exists in Redis. If the key returns `redisNil`, the migration executes and then sets the sentinel to `"true"`. On subsequent startups, the sentinel exists, so the routine exits immediately without redundant scanning or deletion.

### Which cache namespaces does Nitter migrate on startup?

According to the source code, Nitter migrates seven specific namespaces: `flatty` (matching `*:*`), `snappyRss` (matching `rss:*`), and five profile-related namespaces (`userBuckets`, `profileDates`, `profileStats`, `userType`, and `verifiedType`) all matching the `p:*` pattern. Each namespace has its own sentinel key for independent tracking.

### Can I manually run Nitter cache migration for testing?

Yes. Developers can import `redis_cache` and call **`migrate(namespace, pattern)`** directly using `waitFor` or `asyncdispatch`. To force a re-migration, you must first delete the corresponding sentinel key from Redis, as the procedure checks for this key before executing any deletions.