How to Configure Single Machine Thresholds in a Clustered SDS Environment

In SDS (Service Downgrade System), every rate limit, concurrency limit, and exception threshold is defined per physical JVM instance, meaning each node in a cluster independently applies the same point-strategy configuration received via heartbeat from the central admin server.

Didi's SDS is an open-source circuit breaker and degradation framework designed for high-throughput microservices. When operating in a clustered deployment, understanding how single machine thresholds configured in a clustered SDS environment behave is critical—each machine enforces limits locally rather than aggregating metrics across the cluster, providing predictable isolation without cross-node contention.

Understanding Per-Machine Threshold Isolation

SDS treats every deployment unit as an isolated entity. Whether running a single instance or a hundred-node cluster, the thresholds defined in a point-strategy apply strictly to the local JVM process. As documented in the repository's README and implemented throughout the codebase, this design choice prevents cascading failures caused by aggregate counting delays and eliminates network coordination overhead during high-load degradation decisions.

The implication is straightforward: if you configure a visit threshold of 10,000 QPS, every single machine in the cluster allows up to 10,000 requests per second independently, rather than sharing a cluster-wide quota.

The Point-Strategy Data Model

All threshold definitions persist in the sds_point_strategy database table, mapped by the PointStrategyDO Java bean in sds-admin/src/main/java/com/didiglobal/sds/admin/dao/bean/PointStrategyDO.java. This object contains the complete set of configurable limits:

• visitThreshold – Per-machine QPS limit
• concurrentThreshold – Maximum concurrent requests per instance
• exceptionThreshold – Raw exception count threshold
• exceptionRateThreshold – Exception percentage threshold
• timeoutThreshold – Timeout duration threshold (milliseconds)
• timeoutCountThreshold – Raw timeout count threshold
• tokenBucketGeneratedTokensInSecond – Token bucket refill rate per instance
• tokenBucketSize – Token bucket capacity per instance

Configuration Flow from Admin to Local JVM

The lifecycle of a threshold configuration follows a strict pull-based synchronization pattern:

Admin Server Persistence

Operators create or modify strategies via the SDS admin UI or REST API. The PointStrategyController in sds-admin/src/main/java/com/didiglobal/sds/admin/controller/PointStrategyController.java handles HTTP requests and delegates to PointStrategyDao to persist changes to the database. This layer acts as the single source of truth for the entire cluster.

Heartbeat Distribution

Every SDS client emits a heartbeat every 10 seconds through SdsHeartBeatService.updatePointStrategyFromWebServer located in sds-client/src/main/java/com/didiglobal/sds/client/service/SdsHeartBeatService.java. Each heartbeat carries the client's current application group, application name, and local version timestamp.

On the server side, HeartbeatServiceImpl.checkAndGetNewestPointStrategy (in sds-admin/src/main/java/com/didiglobal/sds/admin/service/impl/HeartbeatServiceImpl.java) compares the client's version against the latest database records. If the client's copy is stale, the server returns the full list of applicable point-strategies.

Local Cache Application

Upon receiving the heartbeat response, the client updates its in-memory SdsPointStrategyConfig instance (found in sds-client/src/main/java/com/didiglobal/sds/client/config/SdsPointStrategyConfig.java). The downgrade engine—comprising SdsStrategy and AbstractDowngradeTest classes—queries this local configuration for every request. Because each JVM maintains its own independent copy, enforcement remains strictly single-machine even though the configuration originates from a central source.

Practical Configuration Examples

Creating a Point Strategy via REST API

Use the admin server's REST endpoint to define thresholds for a specific business point:

curl -X POST "http://admin-host:8887/sds/pointStrategy/add" \
     -H "Content-Type: application/json" \
     -d '{
           "appGroupName":"BikeBusinessDepartment",
           "appName":"order",
           "point":"orderCreate",
           "visitThreshold":10000,
           "concurrentThreshold":200,
           "exceptionThreshold":50,
           "exceptionRateThreshold":30,
           "timeoutThreshold":2000,
           "timeoutCountThreshold":20,
           "tokenBucketGeneratedTokensInSecond":5000,
           "tokenBucketSize":10000,
           "downgradeRate":80,
           "downgradeEnable":true
         }'

This payload maps directly to PointStrategyDO fields. Once persisted, all clients receive these values on their next heartbeat cycle.

Direct Database Insertion

For automation or migration scenarios, insert directly into the sds_point_strategy table:

INSERT INTO sds_point_strategy(
  app_group_name, app_name, point,
  visit_threshold, concurrent_threshold,
  exception_threshold, exception_rate_threshold,
  timeout_threshold, timeout_count_threshold,
  token_bucket_generated_tokens_in_second, token_bucket_size,
  downgrade_rate, downgrade_enable, version
) VALUES (
  'BikeBusinessDepartment', 'order', 'orderCreate',
  10000, 200, 50, 30,
  2000, 20,
  5000, 10000,
  80, 1, 1
);

The version column enables the server-side diff mechanism; incrementing it triggers client updates.

Accessing Thresholds in Application Code

Retrieve current thresholds programmatically for custom logic or monitoring:

SdsStrategy strategy = SdsEasyUtil.getStrategy("orderCreate");
long visitThreshold = strategy.getVisitThreshold();           // Local QPS limit
long concurrentThreshold = strategy.getConcurrentThreshold(); // Local concurrency limit

Disabling Specific Thresholds

To disable a particular check, set its value to -1 (represented as -1L in the Java source):

UPDATE sds_point_strategy
SET visit_threshold = -1
WHERE app_group_name = 'BikeBusinessDepartment'
  AND app_name = 'order'
  AND point = 'orderCreate';

The downgrade engine interprets -1 as "unlimited" or "disabled" for that specific dimension.

Summary

• Single-machine enforcement: Every SDS client applies thresholds independently; there is no cluster-wide aggregation of request counts or concurrency levels.
• Centralized storage: All configurations live in the sds_point_strategy table, managed via PointStrategyDO, PointStrategyDao, and exposed through PointStrategyController.
• Heartbeat synchronization: Clients poll the admin server every 10 seconds via SdsHeartBeatService, receiving updates processed by HeartbeatServiceImpl.checkAndGetNewestPointStrategy.
• Local caching: Each JVM maintains its own SdsPointStrategyConfig instance, ensuring sub-millisecond access latency for degradation decisions without network overhead.
• Disabling thresholds: Set any threshold column to -1 to disable that specific protection mechanism for the point.

Frequently Asked Questions

Does SDS aggregate thresholds across cluster nodes?

No. SDS intentionally avoids cross-node aggregation. Each physical instance applies the configured thresholds independently to its own traffic. If you configure a 1,000 QPS limit, every node in a 10-node cluster can theoretically process 1,000 QPS simultaneously, yielding 10,000 QPS aggregate capacity.

How often do clients synchronize thresholds with the admin server?

By default, every 10 seconds. The SdsHeartBeatService initiates a heartbeat request to HeartbeatServiceImpl on this interval. Clients only receive full configuration payloads when their local version timestamp differs from the server's current version, minimizing network traffic.

What happens if the admin server is temporarily unavailable?

Clients continue operating using their last cached configuration stored in SdsPointStrategyConfig. Since all threshold enforcement happens locally against in-memory state, degradation protection remains active even during temporary network partitions or admin server downtime, though configuration updates pause until connectivity restores.

Can different nodes in the same cluster have different threshold values?

While all nodes pull from the same sds_point_strategy record for a given application group and point name, transient states can occur during deployment rollouts or network latency. Eventually, all nodes converge to the same values as they complete heartbeat cycles. The design does not support intentional per-node threshold variance through the standard API; all instances sharing the same application name receive identical configurations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}