How Thunder Provides Candidate Posts for the Home-Mixer

Thunder supplies raw tweet candidates to the Home-Mixer through the ThunderSource implementation, which converts gRPC responses from the Thunder service into internal PostCandidate objects for downstream ranking.

Thunder acts as a downstream service that supplies raw tweet candidates to X's Home-Mixer. In the xai-org/x-algorithm repository, the ThunderSource struct serves as the primary entry point for retrieving these candidates, transforming protocol buffer responses into a standardized internal representation. This article examines the end-to-end flow from request construction to candidate transformation.

The ThunderSource Entry Point

The candidate retrieval process begins in home-mixer/sources/thunder_source.rs with the ThunderSource struct. When the Home-Mixer receives a ScoredPostsQuery, the source method initiates the fetch by constructing a GetInNetworkPostsRequest containing essential query parameters.

The request includes the requesting user ID, the list of accounts the user follows, and configuration values such as ThunderMaxResults to limit the response size. It also accepts exclude_tweet_ids to filter out specific posts and a ThunderAlgorithm parameter that determines the retrieval strategy. These parameter keys are defined in home-mixer/params/param.rs.

Request Routing: CAPI vs. Direct gRPC

Depending on the enable_thunder_capi_home_mixer decider flag, ThunderSource routes requests through one of two paths. When the CAPI client is available, it uses the ThunderCapiClient; otherwise, it falls back to a direct gRPC connection via ThunderClient.

The routing logic in home-mixer/sources/thunder_source.rs selects the transport layer dynamically:

let posts = if let Some(capi) = capi {
    // CAPI-based request
    capi.get_in_network_posts(request).await?.posts
} else {
    // Direct GRPC request
    let channel = self.thunder_client.get_random_channel(cluster)
        .ok_or_else(|| "ThunderSource: no available channel".to_string())?;
    InNetworkPostsServiceClient::new(channel).get_in_network_posts(request).await?.into_inner().posts
};

The underlying protocol definitions for these requests reside in xai_thunder_proto/in_network_posts_service.proto, which defines the InNetworkPostsService gRPC interface used by both transport paths.

Transforming Thunder Posts into Candidates

Once the posts vector arrives from the Thunder service, ThunderSource maps each protobuf structure into a PostCandidate defined in home-mixer/models/candidate.rs. This transformation preserves critical metadata required for ranking and thread reconstruction.

The conversion logic constructs ancestor chains and assigns served types:

// Inside ThunderSource::source – core transformation
let candidates: Vec<PostCandidate> = posts.into_iter().map(|post| {
    let in_reply_to = post.in_reply_to_post_id
        .and_then(|id| u64::try_from(id).ok());

    let conversation = post.conversation_id
        .and_then(|id| u64::try_from(id).ok());

    let mut ancestors = Vec::new();
    if let Some(reply) = in_reply_to {
        ancestors.push(reply);
        if let Some(root) = conversation.filter(|&r| r != reply) {
            ancestors.push(root);
        }
    }

    let served_type = if !query.in_network_only {
        pb::ServedType::ForYouInNetwork
    } else {
        pb::ServedType::RankedFollowing
    };

    PostCandidate {
        tweet_id: post.post_id as u64,
        author_id: post.author_id as u64,
        in_reply_to_tweet_id: in_reply_to,
        retweeted_tweet_id: post.source_post_id.and_then(|id| u64::try_from(id).ok()),
        ancestors,
        served_type: Some(served_type),
        ..Default::default()
    }
}).collect();

This mapping extracts:

• tweet_id and author_id directly from the protobuf
• Optional in_reply_to_tweet_id and retweeted_tweet_id fields
• Ancestor tweet IDs constructed from reply-to and conversation root IDs for thread-level ranking

The served type is set based on the query scope:

• ForYouInNetwork for standard For You timeline candidates
• RankedFollowing when the query restricts results to in-network only

Integration with Downstream Ranking

After transformation, ThunderSource returns a Vec<PostCandidate> to the caller. These candidates feed into the broader ranking architecture, specifically the Phoenix candidate pipeline located in home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs.

The candidates carry thread ancestry information that supports conversation-based ranking decisions. By preserving ancestors derived from conversation_id and in_reply_to_post_id, Thunder enables the Home-Mixer to rank entire threads cohesively rather than treating individual tweets in isolation.

Summary

• ThunderSource in home-mixer/sources/thunder_source.rs serves as the dedicated adapter between the Thunder service and Home-Mixer.
• Requests route through either Thunder CAPI or direct gRPC channels based on the enable_thunder_capi_home_mixer decider flag.
• The service returns raw protobuf posts that get mapped to PostCandidate objects with preserved reply chains and retweet metadata.
• Candidates receive served types of ForYouInNetwork or RankedFollowing depending on query constraints.
• Thread ancestry information enables conversation-aware ranking in downstream pipelines.

Frequently Asked Questions

What is the difference between Thunder CAPI and direct gRPC access?

Thunder CAPI provides a higher-level client interface for retrieving posts, while direct gRPC access uses ThunderClient to obtain a random channel and invoke InNetworkPostsServiceClient directly. The system checks the enable_thunder_capi_home_mixer decider flag at runtime to determine which path to use, defaulting to CAPI when available for improved reliability and metrics aggregation.

How does Thunder handle reply chains and conversations?

During candidate transformation, ThunderSource extracts in_reply_to_post_id and conversation_id from each Thunder post. It constructs an ancestors vector containing the reply-to ID and, if different, the conversation root ID. This ancestry chain allows the Home-Mixer to identify related tweets within a conversation thread and apply thread-level ranking logic.

What parameters control the volume and algorithm of Thunder candidates?

The parameters defined in home-mixer/params/param.rs include ThunderMaxResults, which limits the number of returned posts, and ThunderAlgorithm, which specifies the retrieval strategy. The request also accepts exclude_tweet_ids to prevent specific tweet IDs from appearing in the candidate set, supporting deduplication against previously served content.

Where is the Thunder service implementation located?

While home-mixer/sources/thunder_source.rs contains the client-side integration code, the actual Thunder service implementation resides in thunder/thunder_service.rs. The protocol buffer definitions shared between client and service are defined in xai_thunder_proto/in_network_posts_service.proto, ensuring type consistency across the gRPC boundary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}