SDS Client Reconnection and Data Synchronization: Inside Didi's Service Degrade System

SDS clients maintain fault-tolerant connections using a dual-list URL pool with automatic failover, combined with scheduled heartbeat uploads and periodic strategy pulls to ensure real-time data synchronization with the SDS admin server.

The didi/sds (Service Degrade System) repository provides a robust client-side implementation designed to survive network partitions and server failures while keeping downgrade strategies synchronized. The client employs a sophisticated reconnection mechanism based on URL rotation and a dual-channel communication protocol that separates heartbeat uploads from strategy pulls.

URL Pool Architecture and Load Balancing

When a client initializes through CommonSdsClient → initHeartBeat, the system invokes SdsHeartBeatService.createOnlyOne(...) which calls buildServerUrl【/sds-client/src/main/java/com/didiglobal/sds/client/service/SdsHeartBeatService.java#L29】. This method parses the comma-separated serverAddrList configuration into two distinct pools:

• SERVER_UPLOAD_URL_LIST: Endpoints for posting statistics to sds/heartbeat/add
• SERVER_PULL_URL_LIST: Endpoints for fetching strategies from sds/heartbeat/pullstrategy

Both lists undergo random shuffling during initialization to distribute load across SDS admin servers【L50-L53】. This shuffling ensures that clients in a large deployment don't simultaneously hammer the same server instance.

private void buildServerUrl(String serverAddrList) {
    String[] urlArray = serverAddrList.split(",");
    for (String url : urlArray) {
        if (StringUtils.isBlank(url)) continue;
        url = url.trim();
        SERVER_UPLOAD_URL_LIST.add(url.endsWith("/") ?
                url + UPLOAD_HEARTBEAT_PATH : url + "/" + UPLOAD_HEARTBEAT_PATH);
        SERVER_PULL_URL_LIST.add(url.endsWith("/") ?
                url + PULL_POINT_STRATEGY_PATH : url + "/" + PULL_POINT_STRATEGY_PATH);
    }
    Collections.shuffle(SERVER_UPLOAD_URL_LIST);
    Collections.shuffle(SERVER_PULL_URL_LIST);
}

Automatic Reconnection on Failure

The SDS client implements graceful degradation through URL rotation rather than circuit breakers. When HTTP exceptions occur, the client automatically advances to the next available server URL.

Upload Channel Failover

In uploadHeartbeatData(), the client attempts to post statistics via HttpUtils.post(getCurUploadUrl(), param). If the request throws an exception, the client logs the error and invokes getNextUploadUrl() to rotate the upload index【L24-L31】:

try {
    body = HttpUtils.post(getCurUploadUrl(), param);
} catch (Exception e) {
    String curUrl = getCurUploadUrl();
    String nextUrl = getNextUploadUrl(); // Rotates the list
    logger.warn("Heartbeat upload error, switch from " + curUrl + " to " + nextUrl, e);
}

Pull Channel Failover

Similarly, updatePointStrategyFromWebServer() handles failures in the strategy synchronization channel by calling getNextPullUrl() when exceptions occur【L78-L85】. Both channels maintain independent indices (currentUploadUrlIndex and currentPullUrlIndex) that increment modulo the list size, guaranteeing exhaustive retry across all configured servers.

Data Synchronization Workflow

SDS clients synchronize state through two complementary mechanisms: pushing heartbeat statistics and pulling strategy updates.

Heartbeat Upload Content

Every statistical cycle, the client constructs a HeartbeatRequest containing:

• Application identifiers, IP address, and hostname
• Cycle end time timestamp
• A map of SdsCycleInfo objects tracking visit counts, exception counts, concurrent counts, downgrade counts, and timeout counts per monitored point【L99-L113】

This payload travels to the selected upload URL via HTTP POST.

Strategy Pull and Cache Refresh

Every 5 seconds (CYCLE_BUCKET_NUM * BUCKET_TIME / 2), the CyclePullPointStrategyTask sends a lightweight HeartbeatRequest containing the current local version and list of active points. The server responds with a HeartBeatResponse indicating whether strategies have changed.

When response.isChanged() returns true, the client:

1. Updates the local version timestamp
2. Extracts the SdsSchemeName and strategy list
3. Populates a new ConcurrentHashMap with SdsStrategy objects keyed by point name
4. Refreshes local services:
   • SdsStrategyService.resetAll(strategyMap) updates downgrade thresholds
   • SdsDowngradeReturnValueService.reset(strategyMap) updates custom return values【L92-L100】

if (response.isChanged()) {
    version = response.getVersion();
    Map<String, SdsStrategy> strategies = new ConcurrentHashMap<>();
    for (SdsStrategy s : response.getStrategies()) {
        strategies.put(s.getPoint(), s);
    }
    resetPointInfo(strategies);   // Refreshes StrategyService & ReturnValueService
}

Scheduled Task Execution

The CycleDataService class orchestrates synchronization timing through dedicated ScheduledExecutorService instances initialized in static blocks【L56-L74】.

CycleClearAndUploadTask

This task executes after each half-cycle, clearing the next-cycle counters and triggering sdsHeartBeatService.uploadHeartbeatData()【L48-L55】:

cleanAndUploadExecutor.schedule(new CycleClearAndUploadTask(),
        calDistanceNextExecuteTime(), TimeUnit.MILLISECONDS);

CyclePullPointStrategyTask

Running every 5 seconds, this task ensures rapid propagation of strategy changes from server to client:

pullPointStrategyExecutor.scheduleWithFixedDelay(
        new CyclePullPointStrategyTask(), 0,
        CYCLE_BUCKET_NUM * BUCKET_TIME / 2, TimeUnit.SECONDS);

Both tasks utilize single-threaded executors to maintain deterministic ordering and prevent race conditions during map updates.

Summary

• Dual-list URL pools separate upload and pull traffic while providing load balancing through random shuffling.
• Automatic reconnection occurs via round-robin URL rotation when HTTP errors are detected, ensuring clients survive individual server failures.
• Heartbeat uploads transmit detailed statistics every cycle, while strategy pulls check for updates every 5 seconds.
• Version-based diffing minimizes unnecessary cache refreshes; only changed strategies trigger local service updates.
• Independent indices for upload and pull channels prevent a single failed endpoint from blocking both synchronization directions.

Frequently Asked Questions

How does the SDS client handle complete server cluster outages?

The client rotates through all configured URLs using modulo arithmetic on the index counters. If all servers fail, the client continues retrying from the beginning of the list on subsequent cycles, using the randomized order established at initialization. Local downgrade strategies remain cached in ConcurrentHashMap, allowing the client to continue operating with last-known-good configurations during server outages.

What triggers the client to update its local strategy cache?

The client updates its cache only when HeartBeatResponse.isChanged() returns true, indicating the server's strategy version differs from the client's local version timestamp. This diff-based approach prevents unnecessary object allocation and map updates when strategies remain stable, reducing GC pressure in high-throughput applications.

Can the heartbeat upload and strategy pull intervals be configured independently?

While both intervals derive from CYCLE_BUCKET_NUM and BUCKET_TIME constants defined in CycleDataService, the upload task schedules relative to cycle boundaries using calDistanceNextExecuteTime(), whereas the pull task uses a fixed 5-second delay (CYCLE_BUCKET_NUM * BUCKET_TIME / 2). Modifying these requires changing the source constants and rebuilding the client, as they are not currently exposed as external configuration parameters.

Is the SDS client thread-safe for concurrent point monitoring?

Yes. The client uses ConcurrentHashMap for storing SdsStrategy objects and atomic operations for counter increments. The SdsHeartBeatService methods for URL rotation use synchronized blocks to prevent index corruption during failover scenarios. However, strategy updates occur atomically via map replacement rather than individual entry updates, ensuring readers always see consistent strategy states.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}