# How the Recos Decider Component Is Configured for Dynamic Feature Flagging and Effective Load Shedding

> Learn how to configure the Recos Decider for dynamic feature flagging and load shedding using its dual-layer YAML system. Enable runtime changes without deployments.

- Repository: [X (fka Twitter)/the-algorithm](https://github.com/twitter/the-algorithm)
- Tags: internals
- Published: 2026-03-03

---

**The Recos Decider component leverages a dual-layer YAML configuration system—combining a base file with environment-specific overlays—to enable runtime feature flagging and automated load shedding without requiring code deployments or service restarts.**

The Recos Decider component serves as the central configuration mechanism for Twitter's recommendation pipelines, enabling dynamic feature flagging and effective load shedding through a flexible YAML-driven architecture. Implemented in the `twitter/the-algorithm` repository, this system allows operators to toggle experimental features and shed traffic from hot endpoints instantly by updating overlay configuration files without requiring service restarts or code changes.

## Dynamic Feature Flagging Configuration

### BaseDecider.scala Architecture and YAML Loading

The core abstraction resides in [`src/scala/com/twitter/recos/decider/BaseDecider.scala`](https://github.com/twitter/the-algorithm/blob/main/src/scala/com/twitter/recos/decider/BaseDecider.scala), which initializes the decider by loading two distinct YAML sources. The `RecosDecider` case class extends `BaseDecider` and specifies both a base configuration and an environment-specific overlay:

```scala
case class RecosDecider(env: String, cluster: String = "atla") extends BaseDecider {
  override val baseConfig = Some("/com/twitter/recos/config/decider.yml")
  override val overlayConfig = Some(
    s"/usr/local/config/overlays/recos/service/prod/$cluster/decider_overlay.yml"
  )
}

```

The `baseConfig` contains the complete set of possible feature flags with default rollout percentages, while the `overlayConfig` provides environment-specific overrides that take precedence at runtime.

### Runtime Flag Evaluation with isAvailable Methods

`BaseDecider` exposes three primary methods for runtime checks:

```scala
def isAvailable(feature: String, recipient: Option[Recipient]): Boolean
def isAvailable(feature: String): Boolean
def isAvailableExceptTeam(feature: String, id: Long, isUser: Boolean = true): Boolean

```

The `isAvailableExceptTeam` method enables **team-only overrides**, allowing engineers to enable a flag for predefined internal user IDs while keeping it disabled for production traffic. This facilitates safe internal testing of experimental features.

Typical usage in the Recos codebase checks both the feature flag and team membership:

```scala
val should = decider.isAvailableExceptTeam(
  RecosDecider.recosIncomingTraffic + "_" + displayLocation,
  userId,
  isUser = true
)

```

### Team-Only Overrides for Internal Testing

The `isAvailableExceptTeam` functionality relies on a `TeamUsers` configuration that defines which user IDs constitute the internal team. When this method returns `true`, the feature is active for that specific user regardless of the global rollout percentage, enabling **dark launches** and **canary testing** without affecting the broader user base.

## Effective Load Shedding Implementation

### EndpointLoadShedder.scala Wrapper Pattern

The `EndpointLoadShedder` class in [`src/scala/com/twitter/recos/decider/EndpointLoadShedder.scala`](https://github.com/twitter/the-algorithm/blob/main/src/scala/com/twitter/recos/decider/EndpointLoadShedder.scala) provides a higher-order function pattern that intercepts requests before they reach business logic:

```scala
class EndpointLoadShedder(decider: GraphDecider) {
  private val keyPrefix = "enable_loadshedding"

  def apply[T](endpointName: String)(serve: => Future[T]): Future[T] = {
    val key = s"${keyPrefix}_${decider.graphNamePrefix}_${endpointName}"
    if (decider.isAvailable(key, recipient = Some(RandomRecipient)))
      Future.exception(LoadSheddingException)
    else serve
  }
}

object EndpointLoadShedder {
  object LoadSheddingException extends Exception with NoStackTrace
}

```

When the decider key is enabled, the wrapper throws `LoadSheddingException`, which downstream services catch to return empty responses immediately. This mechanism effectively sheds load by short-circuiting expensive recommendation computations before they execute.

### Load Shedding Keys and Exception Handling

Decider keys for load shedding follow a strict naming convention: `enable_loadshedding_<graph>_<endpoint>`. For example:

- `enable_loadshedding_user-tweet-graph_relatedTweets`
- `enable_loadshedding_user-video-graph_videoGraphTweetBasedRelatedTweets`

These keys accept fractional values (0-100) in the YAML configuration, allowing operators to shed specific percentages of traffic rather than implementing binary on/off switches.

### Integration in Graph Services

The load shedding pattern appears consistently across recommendation services. In [`src/scala/com/twitter/recos/user_video_graph/UserVideoGraph.scala`](https://github.com/twitter/the-algorithm/blob/main/src/scala/com/twitter/recos/user_video_graph/UserVideoGraph.scala):

```scala
override def tweetBasedRelatedTweets(request: TweetBasedRelatedTweetRequest): Future[RelatedTweetResponse] =
  endpointLoadShedder("videoGraphTweetBasedRelatedTweets") {
    tweetBasedRelatedTweetsHandler(request).raiseWithin(defaultTimeout)
  }.rescue {
    case EndpointLoadShedder.LoadSheddingException => Future.value(RelatedTweetResponse())
    case e => 
      log.info("user-video-graph_tweetBasedRelatedTweets" + e)
      Future.value(RelatedTweetResponse())
  }

```

Similar implementations exist in `UserTweetGraph`, `CrMixer`, and `TopicSocialProof` (which uses [`topic-social-proof/server/src/main/scala/com/twitter/tsp/common/LoadShedder.scala`](https://github.com/twitter/the-algorithm/blob/main/topic-social-proof/server/src/main/scala/com/twitter/tsp/common/LoadShedder.scala) for display-location-based shedding), ensuring consistent load management across the entire recommendation stack.

## Configuration File Structure and Overlay System

The decider system employs a two-tier configuration approach that separates default values from runtime overrides. The **base configuration** ([`/com/twitter/recos/config/decider.yml`](https://github.com/twitter/the-algorithm/blob/main//com/twitter/recos/config/decider.yml)) contains the complete schema of available flags:

```yaml
recos_incoming_traffic: 100.0
recos_should_return: 100.0
recos_should_dark: 10.0
enable_loadshedding_user-tweet-graph_relatedTweets: 0.0
enable_loadshedding_user-video-graph_videoGraphTweetBasedRelatedTweets: 0.0

```

The **overlay configuration** (`/usr/local/config/overlays/recos/service/prod/$cluster/decider_overlay.yml`) contains only modified values:

```yaml
recos_should_dark: 50.0
enable_loadshedding_user-tweet-graph_relatedTweets: 30.0

```

When the decider initializes, it merges the overlay onto the base, with overlay values taking precedence. This design allows operators to instantly toggle features, perform gradual rollouts by adjusting percentage values, and enable emergency load shedding during incidents.

## Summary

- The **Recos Decider** component utilizes a **dual-layer YAML configuration** system, combining base files and environment-specific overlays to enable runtime configuration changes without code deployment.
- **Dynamic feature flagging** is implemented through [`BaseDecider.scala`](https://github.com/twitter/the-algorithm/blob/main/BaseDecider.scala), which provides `isAvailable` and `isAvailableExceptTeam` methods for checking flag status and enabling team-only overrides for safe internal testing.
- **Effective load shedding** is achieved via [`EndpointLoadShedder.scala`](https://github.com/twitter/the-algorithm/blob/main/EndpointLoadShedder.scala), a higher-order function wrapper that checks decider keys following the `enable_loadshedding_<graph>_<endpoint>` pattern and throws `LoadSheddingException` to short-circuit requests during high load.
- The configuration architecture separates default values in [`/com/twitter/recos/config/decider.yml`](https://github.com/twitter/the-algorithm/blob/main//com/twitter/recos/config/decider.yml) from runtime overrides in [`/usr/local/config/overlays/.../decider_overlay.yml`](https://github.com/twitter/the-algorithm/blob/main//usr/local/config/overlays/.../decider_overlay.yml), allowing instant operational adjustments to traffic percentages and feature availability.

## Frequently Asked Questions

### What is the difference between the base decider.yml and the overlay configuration?

The base [`decider.yml`](https://github.com/twitter/the-algorithm/blob/main/decider.yml) contains the complete set of default feature flags and their rollout percentages, packaged with the application at build time in [`/com/twitter/recos/config/decider.yml`](https://github.com/twitter/the-algorithm/blob/main//com/twitter/recos/config/decider.yml). The overlay [`decider_overlay.yml`](https://github.com/twitter/the-algorithm/blob/main/decider_overlay.yml) resides on the server filesystem at `/usr/local/config/overlays/recos/service/prod/$cluster/decider_overlay.yml` and contains only modified values that override the base configuration at runtime. This separation allows operators to change feature flags or load-shedding percentages instantly without rebuilding or redeploying the service.

### How does the EndpointLoadShedder determine when to drop requests?

The `EndpointLoadShedder` constructs a decider key using the pattern `enable_loadshedding_<graph>_<endpoint>`, then checks availability via `decider.isAvailable(key, recipient = Some(RandomRecipient))`. If the key is enabled based on the configured percentage, the wrapper throws `LoadSheddingException`, which downstream services catch to return empty responses immediately. This mechanism effectively sheds load by short-circuiting expensive recommendation computations before they execute.

### Can feature flags be enabled for specific internal users only?

Yes, the `BaseDecider` provides the `isAvailableExceptTeam` method specifically for this use case. This method checks if a user ID belongs to a predefined "team" list (configured in `TeamUsers`) and returns true for team members regardless of the global flag percentage. This enables safe internal testing and dark launches of experimental features, allowing engineers to validate changes with internal accounts while keeping the feature disabled for public traffic.

### What files contain the actual implementation of the Recos Decider?

The core implementation resides in [`src/scala/com/twitter/recos/decider/BaseDecider.scala`](https://github.com/twitter/the-algorithm/blob/main/src/scala/com/twitter/recos/decider/BaseDecider.scala), which defines the `BaseDecider` trait and `RecosDecider` case class. The load-shedding wrapper is implemented in [`src/scala/com/twitter/recos/decider/EndpointLoadShedder.scala`](https://github.com/twitter/the-algorithm/blob/main/src/scala/com/twitter/recos/decider/EndpointLoadShedder.scala). Concrete usage examples can be found in service implementations such as [`src/scala/com/twitter/recos/user_video_graph/UserVideoGraph.scala`](https://github.com/twitter/the-algorithm/blob/main/src/scala/com/twitter/recos/user_video_graph/UserVideoGraph.scala) and [`src/scala/com/twitter/recos/user_tweet_graph/UserTweetGraph.scala`](https://github.com/twitter/the-algorithm/blob/main/src/scala/com/twitter/recos/user_tweet_graph/UserTweetGraph.scala).