How Twitter's Algorithm Manages Lifecycle and Stale Data Cleanup in In-Memory Graphs

The Twitter algorithm platform validates every in-memory graph slice against the live Social Graph Service, filters stale edges at query time, and enforces automatic expiration through TTL-based caches, ensuring no stale data persists beyond configurable time windows.

The twitter/the-algorithm repository implements a multi-layered defense against stale data in its real-time recommendation pipelines. Every time the system constructs an in-memory follow-graph or similarity-graph structure, it assumes the underlying snapshot may contain invalid edges. This article examines the specific mechanisms—live validation, TTL eviction, and batch job boundaries—that enforce strict lifecycle management and stale data cleanup for in-memory graphs.

Real-Time Graph Validation and Stale-Edge Filtering

The RealGraphFollowGraphDataProvider component demonstrates the primary defense against stale follow relationships. When a user’s follow list exceeds the maximum threshold served by the primary Social Graph Service (SGS), the system supplements the data from a bulk RealGraph snapshot. However, it never trusts these supplemental edges blindly.

Supplementing Graph Data from RealGraph Snapshots

In timelineranker/server/src/main/scala/com/twitter/timelineranker/visibility/RealGraphFollowGraphDataProvider.scala, the provider first queries SGS. If the result hits the maxFollowingCount, it fetches the full candidate set from the RealGraph client:

if (sgsFollows.size >= maxFollowingCount) {
  // fetch full set from RealGraph
  realGraphClient(Seq(userId))
    .map(_.getOrElse(userId, EmptyRealGraphResponse))
    .map(_.candidates.map(_.userId))
    .flatMap { realGraphFollows =>
      // **Stale‑edge filtering**
      val verifiedRealGraphFollows =
        socialGraphClient.getFollowOverlap(userId, realGraphFollows)

      verifiedRealGraphFollows.map { follows =>
        // merge & deduplicate
        (sgsFollows ++ follows).distinct
      }
    }
}

Cross-Validation Against the Social Graph Service

The critical stale-data cleanup occurs via socialGraphClient.getFollowOverlap. This method validates the RealGraph candidate edges against the live SGS, returning only those follows that still exist. Any edge that has been removed since the snapshot was taken is silently dropped, preventing stale relationships from polluting the in-memory graph.

Observability and Fail-Open Safeguards

The system tracks stale-edge rates through stat counters such as nonOverlappingSizeStat and realGraphEmptyCounter. Additionally, a FailOpenHandler wrapper ensures that if the RealGraph supplement fetch fails, the system gracefully falls back to the original SGS data rather than exposing incomplete or potentially stale graph slices to downstream pipelines.

Batch-Computed Similarity Graphs and Ephemeral Lifecycle

For similarity-based recommendations, the TopUsersSimilarityGraph constructs k-nearest-neighbor (KNN) graphs entirely in memory during batch processing. These structures have a naturally bounded lifecycle that eliminates the risk of long-term stale data accumulation.

In-Memory KNN Graph Construction

In src/scala/com/twitter/simclusters_v2/scalding/TopUsersSimilarityGraph.scala, the pipeline builds weighted neighbor lists on-the-fly:

def topUsersInMemory(k: Int, maxDegree: Int): TypedPipe[(Long, Map[Long, Max[Float]])] = {
  // ... build weighted neighbor lists
  // no explicit “stale” check here – the source data is a fresh snap‑shotted
  // similarity scores are recomputed every batch run, so the graph naturally expires.
}

Natural Expiration via Job Boundaries

Because the graph exists only for the duration of the batch job execution, it naturally expires when the job completes. The similarity scores are recomputed from fresh snapshots every run (hourly or daily), and the in-memory structures are discarded after writing results to downstream stores such as Thrift files or BigQuery tables. This architectural pattern ensures that no stale similarity graph persists beyond a single batch window.

TTL-Based Cache Eviction and Automatic Cleanup

For hot-path data that must persist across requests, the platform relies on TTL-based in-memory caches using Caffeine and Memcached wrappers. These caches enforce automatic lifecycle management through configurable expiration policies.

Configurable Expiration Policies

In tweet-mixer/server/src/main/scala/com/twitter/tweet_mixer/module/InMemoryCacheModule.scala, caches are defined with explicit TTL boundaries:

val cache = Caffeine
  .newBuilder()
  .expireAfterWrite(TTLSeconds, TimeUnit.SECONDS)   // <-- automatic expiry
  .maximumSize(maxSize)
  .build[Key, Value]()

The TTL values are exposed as service flags (e.g., inProcessCacheTtlMs in tweetypie/server/src/main/scala/com/twitter/tweetypie/config/Main.scala), allowing operators to tune cleanup intervals without code changes.

Soft-Expire Handling for Remote Stores

When interacting with remote key-value stores such as Memcached or Strato, the CachingKeyValueRepository layer distinguishes between soft-expired and hard-expired entries. In tweetypie/servo/repo/src/main/scala/com/twitter/servo/repository/CachingKeyValueRepository.scala, the logic handles stale values gracefully:

if (wasntFound || expired.contains(key)) {
  // Treat as a miss and trigger a read‑through fetch
}

Even if a soft-expired value is returned to serve the current request, a background fetch is triggered to refresh the cache, ensuring that subsequent queries receive fresh data and that stale graph edges do not accumulate over time.

Summary

• Live validation – Every supplemental graph slice from RealGraph is cross-checked against the Social Graph Service using getFollowOverlap, dropping stale edges before they reach downstream pipelines.
• Ephemeral batch graphs – Similarity graphs constructed in TopUsersSimilarityGraph exist only for the duration of their batch job, naturally expiring after results are persisted.
• Automatic TTL eviction – Caffeine and Memcached wrappers enforce strict time-based expiration via expireAfterWrite, eliminating long-lived stale entries without manual intervention.
• Soft-expire safety nets – Remote cache repositories return soft-expired values only while triggering background refreshes, preventing stale data from becoming permanent.
• Observability and resilience – Stat counters track stale-edge rates, and FailOpenHandler wrappers ensure that validation failures degrade gracefully rather than exposing corrupt graph data.

Frequently Asked Questions

How does the system prevent stale follow relationships from affecting recommendations?

The system prevents stale follow relationships by validating every edge fetched from the RealGraph snapshot against the live Social Graph Service (SGS). In RealGraphFollowGraphDataProvider.scala, the getFollowOverlap method checks which candidate follows still exist in SGS and silently drops any that have been removed since the snapshot was taken. This ensures only currently valid edges are merged into the final in-memory graph.

What happens to in-memory similarity graphs after the batch job completes?

In-memory similarity graphs are ephemeral by design. Components like TopUsersSimilarityGraph.scala construct k-nearest-neighbor graphs entirely in RAM during a batch Scalding job. Once the job writes the computed results to downstream storage (such as Thrift files or BigQuery), the in-memory structures are discarded. Because the pipeline recomputes these graphs from fresh snapshots on every scheduled run (hourly or daily), no stale similarity data persists between job executions.

How does the TTL-based cache configuration prevent long-term data staleness?

The platform uses Caffeine and Memcached wrappers with explicit time-to-live (TTL) settings to enforce automatic expiration. In InMemoryCacheModule.scala, caches are built with expireAfterWrite(TTLSeconds, TimeUnit.SECONDS), which guarantees that entries are evicted after a configurable duration regardless of access patterns. Operators can tune these intervals via service flags (such as inProcessCacheTtlMs in tweetypie), ensuring that cached graph slices never outlive their freshness window.

What is the difference between soft-expire and hard-expire handling in remote caches?

Soft-expire handling allows the system to serve slightly stale data while refreshing it in the background, whereas hard-expire treats aged entries as complete misses. In CachingKeyValueRepository.scala, when a key is soft-expired, the repository returns the cached value to the caller but triggers an asynchronous read-through fetch to update the store. If the entry is hard-expired or missing, it blocks to fetch fresh data from the canonical source. This hybrid approach prevents cache stampedes while ensuring that graph data eventually converges to the latest state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}