How the Recos Decider Component Is Configured for Dynamic Feature Flagging and Effective Load Shedding

The Recos Decider component leverages a dual-layer YAML configuration system—combining a base file with environment-specific overlays—to enable runtime feature flagging and automated load shedding without requiring code deployments or service restarts.

The Recos Decider component serves as the central configuration mechanism for Twitter's recommendation pipelines, enabling dynamic feature flagging and effective load shedding through a flexible YAML-driven architecture. Implemented in the twitter/the-algorithm repository, this system allows operators to toggle experimental features and shed traffic from hot endpoints instantly by updating overlay configuration files without requiring service restarts or code changes.

Dynamic Feature Flagging Configuration

BaseDecider.scala Architecture and YAML Loading

The core abstraction resides in src/scala/com/twitter/recos/decider/BaseDecider.scala, which initializes the decider by loading two distinct YAML sources. The RecosDecider case class extends BaseDecider and specifies both a base configuration and an environment-specific overlay:

case class RecosDecider(env: String, cluster: String = "atla") extends BaseDecider {
  override val baseConfig = Some("/com/twitter/recos/config/decider.yml")
  override val overlayConfig = Some(
    s"/usr/local/config/overlays/recos/service/prod/$cluster/decider_overlay.yml"
  )
}

The baseConfig contains the complete set of possible feature flags with default rollout percentages, while the overlayConfig provides environment-specific overrides that take precedence at runtime.

Runtime Flag Evaluation with isAvailable Methods

BaseDecider exposes three primary methods for runtime checks:

def isAvailable(feature: String, recipient: Option[Recipient]): Boolean
def isAvailable(feature: String): Boolean
def isAvailableExceptTeam(feature: String, id: Long, isUser: Boolean = true): Boolean

The isAvailableExceptTeam method enables team-only overrides, allowing engineers to enable a flag for predefined internal user IDs while keeping it disabled for production traffic. This facilitates safe internal testing of experimental features.

Typical usage in the Recos codebase checks both the feature flag and team membership:

val should = decider.isAvailableExceptTeam(
  RecosDecider.recosIncomingTraffic + "_" + displayLocation,
  userId,
  isUser = true
)

Team-Only Overrides for Internal Testing

The isAvailableExceptTeam functionality relies on a TeamUsers configuration that defines which user IDs constitute the internal team. When this method returns true, the feature is active for that specific user regardless of the global rollout percentage, enabling dark launches and canary testing without affecting the broader user base.

Effective Load Shedding Implementation

EndpointLoadShedder.scala Wrapper Pattern

The EndpointLoadShedder class in src/scala/com/twitter/recos/decider/EndpointLoadShedder.scala provides a higher-order function pattern that intercepts requests before they reach business logic:

class EndpointLoadShedder(decider: GraphDecider) {
  private val keyPrefix = "enable_loadshedding"

  def apply[T](endpointName: String)(serve: => Future[T]): Future[T] = {
    val key = s"${keyPrefix}_${decider.graphNamePrefix}_${endpointName}"
    if (decider.isAvailable(key, recipient = Some(RandomRecipient)))
      Future.exception(LoadSheddingException)
    else serve
  }
}

object EndpointLoadShedder {
  object LoadSheddingException extends Exception with NoStackTrace
}

When the decider key is enabled, the wrapper throws LoadSheddingException, which downstream services catch to return empty responses immediately. This mechanism effectively sheds load by short-circuiting expensive recommendation computations before they execute.

Load Shedding Keys and Exception Handling

Decider keys for load shedding follow a strict naming convention: enable_loadshedding_<graph>_<endpoint>. For example:

• enable_loadshedding_user-tweet-graph_relatedTweets
• enable_loadshedding_user-video-graph_videoGraphTweetBasedRelatedTweets

These keys accept fractional values (0-100) in the YAML configuration, allowing operators to shed specific percentages of traffic rather than implementing binary on/off switches.

Integration in Graph Services

The load shedding pattern appears consistently across recommendation services. In src/scala/com/twitter/recos/user_video_graph/UserVideoGraph.scala:

override def tweetBasedRelatedTweets(request: TweetBasedRelatedTweetRequest): Future[RelatedTweetResponse] =
  endpointLoadShedder("videoGraphTweetBasedRelatedTweets") {
    tweetBasedRelatedTweetsHandler(request).raiseWithin(defaultTimeout)
  }.rescue {
    case EndpointLoadShedder.LoadSheddingException => Future.value(RelatedTweetResponse())
    case e => 
      log.info("user-video-graph_tweetBasedRelatedTweets" + e)
      Future.value(RelatedTweetResponse())
  }

Similar implementations exist in UserTweetGraph, CrMixer, and TopicSocialProof (which uses topic-social-proof/server/src/main/scala/com/twitter/tsp/common/LoadShedder.scala for display-location-based shedding), ensuring consistent load management across the entire recommendation stack.

Configuration File Structure and Overlay System

The decider system employs a two-tier configuration approach that separates default values from runtime overrides. The base configuration (/com/twitter/recos/config/decider.yml) contains the complete schema of available flags:

recos_incoming_traffic: 100.0
recos_should_return: 100.0
recos_should_dark: 10.0
enable_loadshedding_user-tweet-graph_relatedTweets: 0.0
enable_loadshedding_user-video-graph_videoGraphTweetBasedRelatedTweets: 0.0

The overlay configuration (/usr/local/config/overlays/recos/service/prod/$cluster/decider_overlay.yml) contains only modified values:

recos_should_dark: 50.0
enable_loadshedding_user-tweet-graph_relatedTweets: 30.0

When the decider initializes, it merges the overlay onto the base, with overlay values taking precedence. This design allows operators to instantly toggle features, perform gradual rollouts by adjusting percentage values, and enable emergency load shedding during incidents.

Summary

• The Recos Decider component utilizes a dual-layer YAML configuration system, combining base files and environment-specific overlays to enable runtime configuration changes without code deployment.
• Dynamic feature flagging is implemented through BaseDecider.scala, which provides isAvailable and isAvailableExceptTeam methods for checking flag status and enabling team-only overrides for safe internal testing.
• Effective load shedding is achieved via EndpointLoadShedder.scala, a higher-order function wrapper that checks decider keys following the enable_loadshedding_<graph>_<endpoint> pattern and throws LoadSheddingException to short-circuit requests during high load.
• The configuration architecture separates default values in /com/twitter/recos/config/decider.yml from runtime overrides in /usr/local/config/overlays/.../decider_overlay.yml, allowing instant operational adjustments to traffic percentages and feature availability.

Frequently Asked Questions

What is the difference between the base decider.yml and the overlay configuration?

The base decider.yml contains the complete set of default feature flags and their rollout percentages, packaged with the application at build time in /com/twitter/recos/config/decider.yml. The overlay decider_overlay.yml resides on the server filesystem at /usr/local/config/overlays/recos/service/prod/$cluster/decider_overlay.yml and contains only modified values that override the base configuration at runtime. This separation allows operators to change feature flags or load-shedding percentages instantly without rebuilding or redeploying the service.

How does the EndpointLoadShedder determine when to drop requests?

The EndpointLoadShedder constructs a decider key using the pattern enable_loadshedding_<graph>_<endpoint>, then checks availability via decider.isAvailable(key, recipient = Some(RandomRecipient)). If the key is enabled based on the configured percentage, the wrapper throws LoadSheddingException, which downstream services catch to return empty responses immediately. This mechanism effectively sheds load by short-circuiting expensive recommendation computations before they execute.

Can feature flags be enabled for specific internal users only?

Yes, the BaseDecider provides the isAvailableExceptTeam method specifically for this use case. This method checks if a user ID belongs to a predefined "team" list (configured in TeamUsers) and returns true for team members regardless of the global flag percentage. This enables safe internal testing and dark launches of experimental features, allowing engineers to validate changes with internal accounts while keeping the feature disabled for public traffic.

What files contain the actual implementation of the Recos Decider?

The core implementation resides in src/scala/com/twitter/recos/decider/BaseDecider.scala, which defines the BaseDecider trait and RecosDecider case class. The load-shedding wrapper is implemented in src/scala/com/twitter/recos/decider/EndpointLoadShedder.scala. Concrete usage examples can be found in service implementations such as src/scala/com/twitter/recos/user_video_graph/UserVideoGraph.scala and src/scala/com/twitter/recos/user_tweet_graph/UserTweetGraph.scala.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}