Redis Cache Key Design in Nitter: Namespace Prefixes and Sharded Buckets

Nitter implements a structured, prefix-based Redis cache key scheme that uses short namespace identifiers (such as p:, pid:, and t:) combined with deterministic construction patterns and hash-mod bucketing to organize cached Twitter data efficiently.

The open-source Twitter frontend Nitter, maintained in the zedeus/nitter repository, stores virtually every cached object in Redis under a carefully designed key hierarchy. This Redis cache key design ensures fast lookups, avoids namespace collisions, and keeps related data grouped together while supporting high-throughput caching of user profiles, tweets, lists, and RSS feeds.

Core Principles of the Cache Key Architecture

Nitter’s cache organization follows three fundamental principles: namespace isolation, deterministic key generation, and sharded storage for high-cardinality data sets.

Namespace Prefix Isolation

Every object type in Nitter receives a unique, short prefix that partitions the Redis keyspace. As implemented in src/redis_cache.nim (lines 66-70), these prefixes prevent collisions between different data types:

• p: — User profiles (userKey)
• pid: — User ID buckets (uidKey)
• t: — Individual tweets (tweetKey)
• l: — Twitter lists (listKey)
• rss: — RSS feed outputs
• bc: — Broadcast metadata
• sp: — Audio spaces
• ai: — Account information

This convention allows administrators to inspect cache contents easily using pattern matching (for example, KEYS p:* to list all cached profiles) while ensuring that a username cannot collide with a tweet ID or list identifier.

Deterministic Key Construction

Keys are built from stable identifiers through template functions that concatenate components with colons. In src/redis_cache.nim, the system uses templates like uidKey, userKey, and listKey as the single source of truth for naming.

For RSS feeds, the redisKey procedure in src/routes/rss.nim (lines 13-17) constructs keys by combining the page type, identifier, and cursor:

proc redisKey(page, name, cursor: string): string =
  page & ":" & name & (if cursor.len > 0: ":" & cursor else: "")

This deterministic approach ensures that identical requests always map to the same Redis entry, maximizing cache hit rates across paginated timelines and search results.

Sharded User-ID Buckets

To handle the high volume of username-to-ID mappings without creating millions of individual hash keys, Nitter implements a sharded bucket system. The uidKey template in src/redis_cache.nim (line 66) calculates the bucket using a hash modulo operation:

template uidKey(name: string): string =
  "pid:" & $(hash(name) div 1_000_000)

Each bucket (for example, pid:123) stores a Redis hash where the field is the lowercased username and the value is the numeric Twitter user ID. This keeps individual hash sizes manageable while maintaining O(1) lookup performance for ID resolution.

Serialization and Storage Efficiency

Before writing to Redis, Nitter serializes objects using the flatty library and compresses the result with supersnappy. This compress(toFlatty(data)) pattern, visible in src/redis_cache.nim (lines 86-100), reduces memory footprint and network transfer costs.

The code follows this pattern for all cached entities:

await r.setEx(key, baseCacheTime, compress(toFlatty(userObject)))

Key Patterns by Object Type

User Profiles

Full user objects are stored under the p: prefix using the userKey template:

let profileKey = userKey("someUser")  # Results in "p:someUser"

await r.setEx(profileKey, baseCacheTime, compressedData)

Tweets and Lists

• Tweets: Stored under t:<tweetId> using the tweetKey template
• Lists: Stored under l:<listId> with list members cached separately under cm: and cmm: prefixes

RSS Feed Caching

RSS endpoints generate keys through the redisKey function and prepend the rss: namespace. In src/routes/rss.nim (lines 49-55), pagination cursors become part of the key structure:

let key = "rss:" & query  # Where query contains page, hash, and cursor

await cacheRss(key, rssData)

This design caches each page of results independently, allowing users to paginate through RSS feeds without redundant API calls to Twitter.

Ephemeral and Long-Term Data

• Broadcasts (bc:): Live video metadata with standard TTL
• Audio Spaces (sp:): Running streams with shorter TTLs
• Account Info (ai:): Cached for 24 hours using setEx("ai:" & toLower(name), ttl, data)

TTL and Expiration Strategy

Nitter applies per-object TTLs based on data volatility. The baseCacheTime (approximately one hour) applies to most objects, while list-related data uses the configurable listCacheTime for longer retention. The setEx command is used throughout src/redis_cache.nim (lines 75-78) to ensure automatic expiration and prevent stale data accumulation.

Summary

• Nitter uses single-character prefixes (p:, t:, l:, etc.) to isolate object types in Redis and prevent key collisions.
• User ID resolution employs sharded buckets (pid:<hash_mod_1M>) that store username-to-ID mappings in Redis hashes for scalable lookups.
• All cached objects undergo flatty serialization followed by supersnappy compression before storage to optimize memory usage.
• RSS pagination implements cursor-based key construction (rss:<page>:<hash>:<cursor>) to cache paginated results independently.
• Helper templates in src/redis_cache.nim (uidKey, userKey, tweetKey) provide a single source of truth for key generation across the codebase.

Frequently Asked Questions

How does Nitter prevent cache key collisions between usernames and tweet IDs?

Nitter enforces strict namespace prefixes through the template functions defined in src/redis_cache.nim. User profiles always use the p: prefix (e.g., p:elonmusk), while tweets use t: (e.g., t:1234567890). This prefix isolation ensures that even if a username matches a numeric ID string, they occupy separate Redis keyspaces.

Why does Nitter use hash bucketing for user ID storage instead of direct string keys?

The sharded bucket design (pid:<bucket_number>) prevents the creation of millions of individual Redis keys for username-to-ID mappings. By hashing the username and dividing by 1,000,000, Nitter groups thousands of user mappings into a manageable number of Redis hashes (approximately 1,000 buckets), reducing memory overhead while maintaining constant-time lookup performance through HGET operations.

How does Nitter cache paginated RSS feeds?

RSS cache keys incorporate the pagination cursor into the key name using the redisKey function in src/routes/rss.nim. The format rss:<page>:<identifier>:<cursor> means that page one and page two of the same timeline receive distinct Redis keys. This allows Nitter to serve cached RSS pages instantly without re-fetching upstream data when users paginate through feeds.

What compression algorithm does Nitter use for cached objects?

Nitter uses supersnappy compression on top of flatty serialization. As implemented in src/redis_cache.nim, the code calls compress(toFlatty(data)) before writing to Redis via setEx. This combination provides fast compression/decompression speeds suitable for high-throughput caching while significantly reducing the bytes stored in Redis compared to raw JSON or MessagePack formats.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}