CoSky Client Beat Health Checking: How Service Discovery Keeps Instances Alive

CoSky's Client Beat mechanism uses periodic TTL renewals in Redis to maintain ephemeral service instance registrations, automatically marking instances as unhealthy when heartbeats stop.

CoSky is an open-source service discovery and configuration platform built for high-performance microservices. Its Client Beat health checking mechanism ensures that only live service instances remain discoverable by requiring periodic heartbeats to refresh Redis TTL values, eliminating the need for separate health check endpoints.

How CoSky Client Beat Health Checking Works

The Client Beat mechanism operates as a continuous renewal loop that keeps ephemeral registrations alive in the Redis backend.

Ephemeral Registration and TTL Management

When a service starts, it registers itself as an ephemeral instance via ServiceRegistry.register in ServiceRegistry.kt. Unlike persistent registrations, ephemeral instances have a time-to-live (TTL) value in Redis. If the TTL expires without renewal, Redis automatically removes the instance key, signaling to the discovery client that the service is unhealthy.

The RenewInstanceService Scheduler

The RenewInstanceService class in RenewInstanceService.kt orchestrates the heartbeat schedule. Created by CoSkyAutoServiceRegistrationAutoConfiguration.kt, this service receives renew properties (initialDelay and period) from CoSkyRegistryProperties.kt.

On startup, the service creates a single-threaded reactor scheduler named CoSky-Renew and schedules a periodic task using schedulePeriodically. By default, the first heartbeat occurs after 1 second (initialDelay), with subsequent renewals every 10 seconds (period).

Implementing the Client Beat Renewal Loop

Each scheduled tick executes the renew() method, which iterates through all registered ephemeral instances and refreshes their TTL in Redis.

// RenewInstanceService.kt - Core renewal logic
private fun renew() {
    val instances = serviceRegistry.registeredEphemeralInstances
    Flux.fromIterable(instances.entries)
        .flatMap { (namespacedId, instance) -> 
            serviceRegistry.renew(instance.namespace, instance) 
        }
        .subscribe()
}

The serviceRegistry.renew() call delegates to RedisServiceRegistry.kt, which executes a Lua script atomically:

-- registry_renew.lua (simplified logic)
if redis.call('exists', KEYS[1]) == 1 then
    redis.call('pexpire', KEYS[1], ARGV[1])
    return 1
end
return 0

This atomic operation ensures that TTL renewal is race-condition free, even under high concurrency.

Failure Handling and Instance Removal

When CoSky Client Beat health checking detects failures, the mechanism handles them gracefully without disrupting the scheduler.

If a renew() call fails due to network issues or Redis unavailability, the error is logged but the periodic scheduler continues running. The critical failure point occurs when the instance fails to renew for longer than the TTL duration (typically 30 seconds). At this point, Redis expires the instance key automatically.

Discovery clients using DiscoveryClient poll the registry and remove expired instances from their local service cache. This makes the instance unavailable for load balancing without requiring explicit deregistration calls from the failed service.

Summary

• CoSky Client Beat health checking maintains service liveness through periodic TTL renewals in Redis.
• The RenewInstanceService schedules heartbeats using a dedicated reactor scheduler with configurable initialDelay and period.
• RedisServiceRegistry executes the registry_renew.lua script atomically to refresh instance TTL values.
• Failed heartbeats result in automatic Redis key expiration, triggering discovery clients to remove the instance from available service pools.

Frequently Asked Questions

How does CoSky's Client Beat differ from traditional health check endpoints?

Traditional service discovery often requires separate HTTP health endpoints that the server polls. CoSky's Client Beat combines registration and health checking into a single mechanism where the client actively renews its own ephemeral registration. This reduces network overhead and eliminates the need for exposed health check ports.

What happens if the Redis connection fails during a Client Beat renewal?

If the Redis connection fails during a renewal attempt, the RenewInstanceService logs the error but continues the scheduled heartbeat loop. The instance remains registered until its TTL expires in Redis. Once the TTL expires (typically after multiple missed renewals), Redis automatically removes the key, and discovery clients treat the service as unhealthy.

Can I customize the Client Beat heartbeat interval?

Yes, the heartbeat interval is configurable through CoSkyRegistryProperties.kt. You can adjust the period (default 10 seconds) and initialDelay (default 1 second) properties in your application configuration. Shorter periods provide faster failure detection but increase Redis load, while longer periods reduce overhead but delay failure detection.

Where is the actual TTL refresh logic implemented in CoSky?

The TTL refresh logic resides in RedisServiceRegistry.kt, which executes the registry_renew.lua Lua script atomically in Redis. This script checks for key existence and updates the TTL using pexpire, ensuring race-condition-free renewal even when multiple service instances or threads attempt concurrent operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}