# How Nitter Performs Cache Migration for Outdated Redis Key Patterns

> Learn how Nitter performs Redis cache migration for outdated key patterns. Discover automatic startup checks, non-blocking scans, and batch deletion for efficient cache management.

- Repository: [Zed/nitter](https://github.com/zedeus/nitter)
- Tags: internals
- Published: 2026-08-29

---

**Nitter automatically migrates outdated Redis key patterns at startup by checking sentinel keys, scanning for legacy patterns with non-blocking SCAN operations, and deleting orphaned keys in batches using Redis pipelining.**

Nitter is an alternative Twitter frontend that relies heavily on Redis for caching tweets, user profiles, and RSS feeds. As the codebase evolves, internal cache key naming conventions change between releases, potentially leaving stale keys that waste memory or serve outdated data. The application handles this automatically at boot time through a migration system implemented in `src/redis_cache.nim` designed to clean up legacy Redis key patterns efficiently.

## The Migration Architecture

### Initialization via initRedisPool

When Nitter boots, the `initRedisPool` procedure in `src/redis_cache.nim` establishes the Redis connection pool using configuration values from `src/config.nim`. Immediately after pool creation, it invokes the `migrate` procedure for each known legacy key pattern to ensure the cache remains consistent with the current schema.

### Sentinel Key Pattern

The migration system uses a **sentinel key** to ensure idempotency. Before processing, `migrate` checks if a specific sentinel key exists using `await r.exists(key)`. If found, the migration skips execution. If absent, the procedure proceeds to scan for and delete legacy keys, then creates the sentinel with value `"true"` to prevent future runs. This guarantees the migration executes exactly once per deployment, even across server restarts.

### Non-Blocking Scan and Pipeline Deletion

The `migrate` procedure uses Redis's `SCAN` command (via `r.scan`) with a cursor starting at `0` and a count of `100000` to enumerate keys matching the old pattern without blocking the server. It then initiates a pipeline with `r.startPipelining()`, queues `DEL` commands for each discovered key, sets the sentinel key using `await r.setk(key, "true")`, and flushes the entire pipeline with `await r.flushPipeline()`. This asynchronous approach minimizes round trips and keeps startup latency low.

## Legacy Key Patterns Migrated

Nitter's `initRedisPool` ([lines 49-55](https://github.com/zedeus/nitter/blob/master/src/redis_cache.nim#L49-L55)) invokes migrations for several historic key families:

- **flatty**: Pattern `*:*` — Removes legacy flatty-encoded objects
- **snappyRss**: Pattern `rss:*` — Cleans RSS feeds stored with previous compression schemes
- **userBuckets**: Pattern `p:*` — Removes per-user bucket structures
- **profileDates**: Pattern `p:*` — Removes cached profile date information
- **profileStats**: Pattern `p:*` — Removes profile statistics
- **userType**: Pattern `p:*` — Removes cached user-type flags
- **verifiedType**: Pattern `p:*` — Removes verified-status flags

## Code Implementation Examples

### Automatic Migration at Startup

During normal server initialization in `src/nitter.nim`, the migration triggers automatically through `initRedisPool`:

```nim

# Load configuration from src/config.nim

let cfg = Config.load()

# Initialize pool and run migrations defined in src/redis_cache.nim

await initRedisPool(cfg)

```

### Adding a New Migration

To handle future key pattern changes, add a new `migrate` call inside `initRedisPool` before the existing calls:

```nim

# In src/redis_cache.nim, append to the initialization sequence:

await migrate("newFeatureCache", "oldfeat:*")

```

### Manual Migration for Testing

For debugging or forced cleanup, call the `migrate` procedure directly with a custom sentinel:

```nim

# Direct invocation to force specific pattern deletion

await migrate("oldCacheMarker", "oldcache:*")

```

## Summary

- Nitter handles Redis schema changes automatically at startup through the `initRedisPool` procedure in `src/redis_cache.nim`.
- The **sentinel key pattern** ensures migrations run only once per deployment, preventing redundant operations across restarts.
- Legacy keys are discovered using **non-blocking SCAN** with a cursor and count of `100000`, then deleted efficiently via **Redis pipelining**.
- Multiple historic patterns are handled, including `*:*` for flatty encoding and `p:*` for various user-related caches like `userBuckets` and `profileStats`.
- The entire process is asynchronous, ensuring minimal impact on server startup time even with large Redis datasets.

## Frequently Asked Questions

### How does Nitter prevent migrations from running on every restart?

The system checks for a specific **sentinel key** before executing each migration. If the sentinel exists, the procedure returns immediately. Only after successfully deleting all legacy keys and flushing the pipeline does the code execute `await r.setk(key, "true")` to create the sentinel. This guarantees idempotent execution across server restarts.

### What happens if a migration fails halfway through?

Because the sentinel key is only set after the pipeline successfully flushes all `DEL` commands, a failed or interrupted migration will not create the sentinel. On the next startup, Nitter will detect the missing sentinel and retry the migration from the beginning, ensuring eventual consistency without manual intervention.

### Why does Nitter use SCAN instead of KEYS for finding legacy patterns?

The `migrate` procedure uses `SCAN` (via `r.scan`) with a cursor-based approach instead of `KEYS` because `KEYS` blocks the Redis server while iterating over the entire keyspace. `SCAN` provides a non-blocking alternative that iterates incrementally using a cursor starting at `0` and a large count parameter (`100000`), preventing performance degradation during startup on large Redis instances.

### Where are the migration calls configured in the codebase?

All migration invocations are hardcoded in the `initRedisPool` procedure between lines 49-55 of `src/redis_cache.nim`. Each call specifies a unique sentinel identifier and the glob pattern to match, such as `await migrate("flatty", "*:*")` for removing legacy flatty-encoded objects or `await migrate("snappyRss", "rss:*")` for old RSS feed compression schemes.