# How Twitter's Algorithm Manages Lifecycle and Stale Data Cleanup in In-Memory Graphs

> Explore how Twitter's algorithm manages in-memory graph lifecycle and stale data cleanup. Discover TTL caches, query-time filtering, and live validation, ensuring data accuracy.

- Repository: [X (fka Twitter)/the-algorithm](https://github.com/twitter/the-algorithm)
- Tags: internals
- Published: 2026-03-03

---

**The Twitter algorithm platform validates every in-memory graph slice against the live Social Graph Service, filters stale edges at query time, and enforces automatic expiration through TTL-based caches, ensuring no stale data persists beyond configurable time windows.**

The `twitter/the-algorithm` repository implements a multi-layered defense against stale data in its real-time recommendation pipelines. Every time the system constructs an in-memory follow-graph or similarity-graph structure, it assumes the underlying snapshot may contain invalid edges. This article examines the specific mechanisms—live validation, TTL eviction, and batch job boundaries—that enforce strict lifecycle management and stale data cleanup for in-memory graphs.

## Real-Time Graph Validation and Stale-Edge Filtering

The **RealGraphFollowGraphDataProvider** component demonstrates the primary defense against stale follow relationships. When a user’s follow list exceeds the maximum threshold served by the primary Social Graph Service (SGS), the system supplements the data from a bulk **RealGraph** snapshot. However, it never trusts these supplemental edges blindly.

### Supplementing Graph Data from RealGraph Snapshots

In [`timelineranker/server/src/main/scala/com/twitter/timelineranker/visibility/RealGraphFollowGraphDataProvider.scala`](https://github.com/twitter/the-algorithm/blob/main/timelineranker/server/src/main/scala/com/twitter/timelineranker/visibility/RealGraphFollowGraphDataProvider.scala), the provider first queries SGS. If the result hits the `maxFollowingCount`, it fetches the full candidate set from the RealGraph client:

```scala
if (sgsFollows.size >= maxFollowingCount) {
  // fetch full set from RealGraph
  realGraphClient(Seq(userId))
    .map(_.getOrElse(userId, EmptyRealGraphResponse))
    .map(_.candidates.map(_.userId))
    .flatMap { realGraphFollows =>
      // **Stale‑edge filtering**
      val verifiedRealGraphFollows =
        socialGraphClient.getFollowOverlap(userId, realGraphFollows)

      verifiedRealGraphFollows.map { follows =>
        // merge & deduplicate
        (sgsFollows ++ follows).distinct
      }
    }
}

```

### Cross-Validation Against the Social Graph Service

The critical stale-data cleanup occurs via `socialGraphClient.getFollowOverlap`. This method validates the RealGraph candidate edges against the live SGS, returning only those follows that still exist. Any edge that has been removed since the snapshot was taken is silently dropped, preventing stale relationships from polluting the in-memory graph.

### Observability and Fail-Open Safeguards

The system tracks stale-edge rates through stat counters such as `nonOverlappingSizeStat` and `realGraphEmptyCounter`. Additionally, a **FailOpenHandler** wrapper ensures that if the RealGraph supplement fetch fails, the system gracefully falls back to the original SGS data rather than exposing incomplete or potentially stale graph slices to downstream pipelines.

## Batch-Computed Similarity Graphs and Ephemeral Lifecycle

For similarity-based recommendations, the **TopUsersSimilarityGraph** constructs **k-nearest-neighbor (KNN) graphs** entirely in memory during batch processing. These structures have a naturally bounded lifecycle that eliminates the risk of long-term stale data accumulation.

### In-Memory KNN Graph Construction

In [`src/scala/com/twitter/simclusters_v2/scalding/TopUsersSimilarityGraph.scala`](https://github.com/twitter/the-algorithm/blob/main/src/scala/com/twitter/simclusters_v2/scalding/TopUsersSimilarityGraph.scala), the pipeline builds weighted neighbor lists on-the-fly:

```scala
def topUsersInMemory(k: Int, maxDegree: Int): TypedPipe[(Long, Map[Long, Max[Float]])] = {
  // ... build weighted neighbor lists
  // no explicit “stale” check here – the source data is a fresh snap‑shotted
  // similarity scores are recomputed every batch run, so the graph naturally expires.
}

```

### Natural Expiration via Job Boundaries

Because the graph exists only for the duration of the batch job execution, it naturally expires when the job completes. The similarity scores are recomputed from fresh snapshots every run (hourly or daily), and the in-memory structures are discarded after writing results to downstream stores such as Thrift files or BigQuery tables. This architectural pattern ensures that no stale similarity graph persists beyond a single batch window.

## TTL-Based Cache Eviction and Automatic Cleanup

For hot-path data that must persist across requests, the platform relies on **TTL-based in-memory caches** using Caffeine and Memcached wrappers. These caches enforce automatic lifecycle management through configurable expiration policies.

### Configurable Expiration Policies

In [`tweet-mixer/server/src/main/scala/com/twitter/tweet_mixer/module/InMemoryCacheModule.scala`](https://github.com/twitter/the-algorithm/blob/main/tweet-mixer/server/src/main/scala/com/twitter/tweet_mixer/module/InMemoryCacheModule.scala), caches are defined with explicit TTL boundaries:

```scala
val cache = Caffeine
  .newBuilder()
  .expireAfterWrite(TTLSeconds, TimeUnit.SECONDS)   // <-- automatic expiry
  .maximumSize(maxSize)
  .build[Key, Value]()

```

The TTL values are exposed as service flags (e.g., `inProcessCacheTtlMs` in [`tweetypie/server/src/main/scala/com/twitter/tweetypie/config/Main.scala`](https://github.com/twitter/the-algorithm/blob/main/tweetypie/server/src/main/scala/com/twitter/tweetypie/config/Main.scala)), allowing operators to tune cleanup intervals without code changes.

### Soft-Expire Handling for Remote Stores

When interacting with remote key-value stores such as Memcached or Strato, the **CachingKeyValueRepository** layer distinguishes between soft-expired and hard-expired entries. In [`tweetypie/servo/repo/src/main/scala/com/twitter/servo/repository/CachingKeyValueRepository.scala`](https://github.com/twitter/the-algorithm/blob/main/tweetypie/servo/repo/src/main/scala/com/twitter/servo/repository/CachingKeyValueRepository.scala), the logic handles stale values gracefully:

```scala
if (wasntFound || expired.contains(key)) {
  // Treat as a miss and trigger a read‑through fetch
}

```

Even if a soft-expired value is returned to serve the current request, a background fetch is triggered to refresh the cache, ensuring that subsequent queries receive fresh data and that stale graph edges do not accumulate over time.

## Summary

- **Live validation** – Every supplemental graph slice from RealGraph is cross-checked against the Social Graph Service using `getFollowOverlap`, dropping stale edges before they reach downstream pipelines.
- **Ephemeral batch graphs** – Similarity graphs constructed in `TopUsersSimilarityGraph` exist only for the duration of their batch job, naturally expiring after results are persisted.
- **Automatic TTL eviction** – Caffeine and Memcached wrappers enforce strict time-based expiration via `expireAfterWrite`, eliminating long-lived stale entries without manual intervention.
- **Soft-expire safety nets** – Remote cache repositories return soft-expired values only while triggering background refreshes, preventing stale data from becoming permanent.
- **Observability and resilience** – Stat counters track stale-edge rates, and `FailOpenHandler` wrappers ensure that validation failures degrade gracefully rather than exposing corrupt graph data.

## Frequently Asked Questions

### How does the system prevent stale follow relationships from affecting recommendations?

The system prevents stale follow relationships by validating every edge fetched from the RealGraph snapshot against the live Social Graph Service (SGS). In [`RealGraphFollowGraphDataProvider.scala`](https://github.com/twitter/the-algorithm/blob/main/RealGraphFollowGraphDataProvider.scala), the `getFollowOverlap` method checks which candidate follows still exist in SGS and silently drops any that have been removed since the snapshot was taken. This ensures only currently valid edges are merged into the final in-memory graph.

### What happens to in-memory similarity graphs after the batch job completes?

In-memory similarity graphs are ephemeral by design. Components like [`TopUsersSimilarityGraph.scala`](https://github.com/twitter/the-algorithm/blob/main/TopUsersSimilarityGraph.scala) construct k-nearest-neighbor graphs entirely in RAM during a batch Scalding job. Once the job writes the computed results to downstream storage (such as Thrift files or BigQuery), the in-memory structures are discarded. Because the pipeline recomputes these graphs from fresh snapshots on every scheduled run (hourly or daily), no stale similarity data persists between job executions.

### How does the TTL-based cache configuration prevent long-term data staleness?

The platform uses Caffeine and Memcached wrappers with explicit time-to-live (TTL) settings to enforce automatic expiration. In [`InMemoryCacheModule.scala`](https://github.com/twitter/the-algorithm/blob/main/InMemoryCacheModule.scala), caches are built with `expireAfterWrite(TTLSeconds, TimeUnit.SECONDS)`, which guarantees that entries are evicted after a configurable duration regardless of access patterns. Operators can tune these intervals via service flags (such as `inProcessCacheTtlMs` in `tweetypie`), ensuring that cached graph slices never outlive their freshness window.

### What is the difference between soft-expire and hard-expire handling in remote caches?

Soft-expire handling allows the system to serve slightly stale data while refreshing it in the background, whereas hard-expire treats aged entries as complete misses. In [`CachingKeyValueRepository.scala`](https://github.com/twitter/the-algorithm/blob/main/CachingKeyValueRepository.scala), when a key is soft-expired, the repository returns the cached value to the caller but triggers an asynchronous read-through fetch to update the store. If the entry is hard-expired or missing, it blocks to fetch fresh data from the canonical source. This hybrid approach prevents cache stampedes while ensuring that graph data eventually converges to the latest state.