Troubleshooting Common Service Discovery Issues in CoSky: A Complete Guide

Most CoSky service discovery failures stem from namespace mismatches, Redis connectivity issues, or TTL/renewal misconfigurations that can be diagnosed by inspecting Lua script execution logs and verifying NamespacedContext alignment.

CoSky is a high-performance, Redis-based distributed service discovery system developed in the ahoo-wang/cosky repository. When microservices fail to register, disappear unexpectedly, or return stale instance lists, developers need systematic troubleshooting approaches. This guide explains how to diagnose and resolve common service discovery issues by analyzing the core components, Lua scripts, and configuration properties that govern CoSky's behavior.

Understanding CoSky's Service Discovery Architecture

CoSky implements service discovery through a layered architecture that separates the API contract from Redis-specific implementations.

The ServiceDiscovery interface in cosky-discovery/src/main/kotlin/me/ahoo/cosky/discovery/ServiceDiscovery.kt defines the contract for querying service instances. The RedisServiceDiscovery class in cosky-discovery/src/main/kotlin/me/ahoo/cosky/discovery/redis/RedisServiceDiscovery.kt executes Lua scripts against Redis to read service topology and instance data atomically.

For Spring Cloud integration, CoSkyDiscoveryClient in cosky-spring-cloud-starter-discovery/src/main/kotlin/me/ahoo/cosky/discovery/spring/cloud/discovery/CoSkyDiscoveryClient.kt exposes the discovery API as a standard Spring DiscoveryClient. The reactive equivalent, CoSkyReactiveDiscoveryClient, handles non-blocking service lookups.

On the registration side, RedisServiceRegistry in cosky-discovery/src/main/kotlin/me/ahoo/cosky/discovery/redis/RedisServiceRegistry.kt persists service instances, handles TTL renewal, and manages deregistration through atomic Lua scripts like registry_register.lua and registry_renew.lua.

Common Service Discovery Failure Scenarios

Service Instances Never Appear in Discovery

When a service instance never appears in the CoSky dashboard or DiscoveryClient results, the registration likely failed or the query uses a different namespace. Check for RedisConnectionException in service logs, which indicates the RedisServiceRegistry cannot reach Redis. Verify the namespace used by the service provider matches the client configuration by comparing NamespacedContext.namespace with CoSkyDiscoveryProperties.namespace. Enable TRACE logging for me.ahoo.cosky.discovery to see the outcome of registry_register.lua execution.

Instances Disappear After a Few Seconds

Intermittent instance availability typically indicates TTL expiration without successful renewal. Inspect RegistryProperties.renewInterval (default 30 seconds) and ensure it is significantly lower than the TTL value. Confirm the renewal task is scheduled by looking for periodic log lines containing Renew instance. Check that registry_renew.lua returns 1 (success) by enabling DEBUG logging, which prints the raw Lua script response.

Stale Metadata or Duplicate Instance IDs

When instance metadata shows old versions or incorrect IP addresses, the issue usually stems from non-unique instance IDs or cached metadata. Ensure each instance generates a unique identifier using NamespacedInstanceId.generate(), which is the default implementation. Verify the metadata map passed to RedisServiceRegistry.register() contains current information. Use the CoSky dashboard to view raw Redis hash entries via service_topology_get.lua and confirm the stored data matches expectations.

DiscoveryClient Returns Empty Lists

When DiscoveryClient.getInstances() returns an empty list for a known service, namespace mismatches or service ID typos are the likely culprits. Call the low-level API directly to bypass Spring abstraction:

val instances = serviceDiscovery.getInstances("myNs", "my-service")

Compare the namespace with NamespacedContext.namespace in the provider. Use Redis CLI to inspect keys directly: SCAN 0 MATCH *myNs:my-service*.

Load Balancer Selects Unhealthy Instances

If the load balancer routes traffic to failed instances, the health statistics may not be updating or weights are misconfigured. Verify ServiceStat updates via service_stat.lua after each request by checking for stat.increment() calls in the service code. Ensure a LoadBalancer implementation such as BinaryWeightRandomLoadBalancer is wired through CoSkyDiscoveryAutoConfiguration.

Step-by-Step Troubleshooting Guide

1. Validate Redis health

   redis-cli ping
   redis-cli info persistence

   Errors indicate connectivity problems preventing both registration and discovery.

2. Check registration scripts

   Look for log entries from RedisServiceRegistry executing registry_register.lua. If missing, the service may have thrown an exception while building the Instance object, such as missing host configuration.

3. Confirm namespace alignment

   println("Provider namespace: ${NamespacedContext.namespace}")
   println("Client namespace: ${coSkyDiscoveryProperties.namespace}")

   Mismatched namespaces are the most frequent source of empty discovery results.

4. Inspect TTL and renewal

   val ttl = redisTemplate.opsForValue().getTimeout("cosky:registry:${instanceId}")
   println("TTL = $ttl seconds")

   If TTL approaches zero and renewal log lines stop appearing, the renewal task in RedisServiceRegistry is failing.

5. Examine Lua script results

   Enable debug logging:

   logging:
     level:
       me.ahoo.cosky.discovery: DEBUG

   The logs print raw Lua script responses (e.g., 1 for success, 0 for failure) from registry_renew.lua and service_topology_get.lua.

6. Use the CoSky dashboard

   The dashboard visualizes the service topology stored in Redis. Missing nodes indicate registration failures; stale nodes indicate TTL expiration without renewal.

7. Run unit tests

   ./gradlew :cosky-discovery:test

   Tests such as RedisServiceDiscoveryTest and RedisServiceRegistryTest exercise the full registration-discovery flow and surface configuration gaps.

Diagnostic Code Examples

Querying Services Programmatically

import me.ahoo.cosky.discovery.ServiceDiscovery
import me.ahoo.cosky.discovery.ServiceInstance
import org.springframework.stereotype.Component

@Component
class ServiceLookup(private val discovery: ServiceDiscovery) {

    fun listInstances(namespace: String, serviceId: String): List<ServiceInstance> {
        // Synchronous API – returns a List
        return discovery.getInstances(namespace, serviceId).collectList().block()!!
    }
}

Uses the ServiceDiscovery interface defined in [ServiceDiscovery.kt](https://github.com/ahoo-wang/cosky/blob/main/cosky-discovery/src/main/kotlin/me/ahoo/cosky/discovery/ServiceDiscovery.kt).

Spring Cloud Discovery Client

import org.springframework.cloud.client.discovery.DiscoveryClient;
import org.springframework.stereotype.Service;

@Service
public class SpringDiscovery {
    private final DiscoveryClient client;

    public SpringDiscovery(DiscoveryClient client) {
        this.client = client;
    }

    public List<ServiceInstance> getInstances(String serviceId) {
        // The client is automatically wired to CoSkyDiscoveryClient
        return client.getInstances(serviceId);
    }
}

The bean is created by [CoSkyDiscoveryClientConfiguration.kt](https://github.com/ahoo-wang/cosky/blob/main/cosky-spring-cloud-starter-discovery/src/main/kotlin/me/ahoo/cosky/discovery/spring/cloud/discovery/CoSkyDiscoveryClientConfiguration.kt).

Manual Service Registration

import me.ahoo.cosky.discovery.ServiceRegistry
import me.ahoo.cosky.discovery.ServiceInstance
import me.ahoo.cosky.discovery.Instance
import me.ahoo.cosky.discovery.namespaced.NamespacedInstanceId

val instance = Instance(
    instanceId = NamespacedInstanceId.generate(),
    serviceId = "order-service",
    host = "10.0.1.12",
    port = 8080,
    metadata = mapOf("version" to "1.2.3")
)

serviceRegistry.register(ServiceInstance.of(instance))

Calls the register method in [RedisServiceRegistry.kt](https://github.com/ahoo-wang/cosky/blob/main/cosky-discovery/src/main/kotlin/me/ahoo/cosky/discovery/redis/RedisServiceRegistry.kt) which runs registry_register.lua.

Configuring TTL and Renewal

cosky:
  discovery:
    enabled: true
    namespace: demo
  registry:
    renew:
      enabled: true
      interval: 15s   # default is 30s

    ttl: 45s           # TTL for each instance key

Properties are bound to [CoSkyDiscoveryProperties.kt](https://github.com/ahoo-wang/cosky/blob/main/cosky-spring-cloud-starter-discovery/src/main/kotlin/me/ahoo/cosky/discovery/spring/cloud/discovery/CoSkyDiscoveryProperties.kt) and [RegistryProperties.kt](https://github.com/ahoo-wang/cosky/blob/main/cosky-discovery/src/main/kotlin/me/ahoo/cosky/discovery/RegistryProperties.kt).

Summary

• Namespace mismatches between providers and consumers are the leading cause of empty discovery results; always verify NamespacedContext.namespace alignment.
• Redis connectivity issues manifest as registration failures or missing Lua script execution logs; validate with redis-cli ping before investigating application code.
• TTL expiration causes intermittent instance disappearance; ensure RegistryProperties.renewInterval is significantly lower than the TTL value and that renewal tasks are scheduled.
• Lua script debugging reveals atomic operation failures; enable DEBUG logging for me.ahoo.cosky.discovery to view raw script responses.
• Unique instance IDs prevent metadata collisions; use NamespacedInstanceId.generate() or ensure manual IDs are globally unique.

Frequently Asked Questions

Why is my service not showing up in CoSky discovery?

The most common cause is a namespace mismatch between the service provider and consumer. Verify that cosky.discovery.namespace is identical in both application.yaml files. Additionally, check for RedisConnectionException in logs, which indicates the RedisServiceRegistry cannot execute registry_register.lua due to connectivity issues.

Why do registered services disappear after a few seconds?

This typically indicates TTL expiration without successful renewal. Check that cosky.registry.renew.enabled is set to true and that cosky.registry.renew.interval (default 30s) is significantly lower than cosky.registry.ttl. Enable DEBUG logging for me.ahoo.cosky.discovery to verify that registry_renew.lua returns 1 (success) periodically.

How do I fix namespace mismatches in CoSky?

Align the namespace configuration across all services by setting cosky.discovery.namespace to the same value (e.g., production or demo) in every application.yaml. Programmatically, you can verify the active namespace by printing NamespacedContext.namespace in the provider and CoSkyDiscoveryProperties.namespace in the consumer to ensure they match before calling getInstances().

How can I debug Lua script failures in CoSky?

Enable DEBUG level logging for the package me.ahoo.cosky.discovery in your application.yaml. This configuration prints the raw responses from Lua scripts such as registry_register.lua, registry_renew.lua, and service_topology_get.lua. A return value of 0 indicates failure (e.g., instance already exists or renewal missed), while 1 indicates success. Combine these logs with Redis CLI commands like EVALSHA to test scripts manually against your Redis instance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}