# How CoCache Ensures Cache Consistency Across Multiple Instances

> Learn how CoCache ensures cache consistency across multiple instances using distributed L1 cache, local L2 cache, and event bus broadcasts for efficient invalidation.

- Repository: [Ahoo Wang/cocache](https://github.com/ahoo-wang/cocache)
- Tags: internals
- Published: 2026-02-23

---

**CoCache ensures cache consistency across multiple instances by combining a distributed L1 cache (e.g., Redis), a local in-process L2 cache, and a publish/subscribe eviction event bus that broadcasts invalidation events to all instances whenever data changes.**

The `ahoo-wang/cocache` library implements an **eventual consistency** model for JVM-based applications running in clustered environments. By tiering cache access and propagating invalidation events through a lightweight event bus, CoCache allows each instance to maintain fast local reads while guaranteeing that stale data is evicted cluster-wide within milliseconds.

## The Three-Layer Consistency Architecture

CoCache achieves **cache consistency across multiple instances** through three coordinated layers:

### Distributed Cache (L1)

The **L1 layer** serves as the authoritative shared backing store, typically backed by Redis. This layer holds the single source of truth for all cache entries across the cluster. When an instance writes data, it updates L1 first, ensuring that any other instance can retrieve the fresh value directly from the distributed store if their local cache misses.

### Client-Side Cache (L2)

The **L2 layer** is a high-speed, in-process map (such as Caffeine or Guava Cache) local to each JVM instance. This layer provides microsecond-level read latency for hot data. However, because each instance maintains its own isolated L2, CoCache must actively invalidate these entries when remote instances perform updates.

### The Eviction Event Bus

The **cache-eviction event bus** is the coordination mechanism that ties the layers together. Implemented in [`GuavaCacheEvictedEventBus.kt`](https://github.com/ahoo-wang/cocache/blob/main/GuavaCacheEvictedEventBus.kt), this component uses Guava’s `EventBus` to publish and subscribe to `CacheEvictedEvent` objects. When one instance mutates a key, it publishes an event; all other instances consume this event and evict the corresponding entry from their L2 caches.

## Write Operations and Event Propagation

All write operations in [`DefaultCoherentCache.kt`](https://github.com/ahoo-wang/cocache/blob/main/DefaultCoherentCache.kt) follow a strict protocol to maintain consistency. Located at [`cocache-core/src/main/kotlin/me/ahoo/cache/consistency/DefaultCoherentCache.kt`](https://github.com/ahoo-wang/cocache/blob/main/cocache-core/src/main/kotlin/me/ahoo/cache/consistency/DefaultCoherentCache.kt), this class implements the core logic for the `setCache`, `evict`, and event handling methods.

### Setting and Evicting Keys

When an instance calls `setCache()` (lines 42–49), the implementation performs three atomic steps:

1. Writes the value to the local L2 cache for immediate visibility
2. Writes the value to the distributed L1 cache for persistence
3. Publishes a `CacheEvictedEvent` to the bus to invalidate peer L2 entries

The `evict()` method (lines 51–56) follows a similar pattern, removing the entry from both L1 and L2 before publishing the eviction event:

```kotlin
// DefaultCoherentCache.kt – evict logic (simplified)
override fun evict(key: K) {
    clientSideCache.evict(key)    // Clear local L2
    distributedCache.evict(key)   // Clear shared L1
    publishEvictEvent(key)        // Notify other instances
}

```

### Handling Remote Eviction Events

Each cache instance registers an `@Subscribe` annotated handler named `onEvicted()` (lines 58–73). When the event bus delivers a `CacheEvictedEvent`, this method checks if the event originated from a different `clientId`. If so, it evicts only the local L2 entry, leaving the L1 data intact:

```kotlin
// DefaultCoherentCache.kt – event handler
@Subscribe
override fun onEvicted(cacheEvictedEvent: CacheEvictedEvent) {
    if (cacheEvictedEvent.clientId == clientId) {
        return // Ignore self-generated events
    }
    clientSideCache.evict(cacheEvictedEvent.key)
}

```

## Read Operations with Cache Stampede Protection

The `getCache()` method (lines 88–135) in [`DefaultCoherentCache.kt`](https://github.com/ahoo-wang/cocache/blob/main/DefaultCoherentCache.kt) implements a cache-aside pattern with **cache stampede protection**. When a key is not found in L2 or L1, the implementation acquires a fine-grained lock from `keyLocks` to ensure that only one thread per instance loads the value from the underlying data source.

After loading, the method populates both L1 and L2 and publishes an eviction event. This guarantees that other instances will invalidate their stale L2 entries before the next read cycle:

```kotlin
// DefaultCoherentCache.kt – getCache with stampede protection
override fun getCache(key: K): CacheValue<V>? {
    // 1. Check L2
    clientSideCache.get(key)?.let { return it }
    
    // 2. Check L1
    distributedCache.getCache(key)?.let { 
        clientSideCache.set(key, it) // Backfill L2
        return it 
    }
    
    // 3. Acquire lock and load from source
    return keyLocks.lock(key) {
        // Load from data source...
        // Write to L1 and L2
        // Publish eviction event for consistency
    }
}

```

## Configuring the Event Bus

The concrete implementation `GuavaCacheEvictedEventBus` wraps Guava’s `EventBus` to provide the `publish()`, `register()`, and `unregister()` methods. This class is defined in [`cocache-core/src/main/kotlin/me/ahoo/cache/consistency/GuavaCacheEvictedEventBus.kt`](https://github.com/ahoo-wang/cocache/blob/main/cocache-core/src/main/kotlin/me/ahoo/cache/consistency/GuavaCacheEvictedEventBus.kt):

```kotlin
// GuavaCacheEvictedEventBus.kt
class GuavaCacheEvictedEventBus : CacheEvictedEventBus {
    private val eventBus = EventBus()
    
    override fun publish(event: CacheEvictedEvent) {
        eventBus.post(event)
    }
    
    override fun register(subscriber: CacheEvictedSubscriber) {
        eventBus.register(subscriber)
    }
}

```

In production deployments, you can replace this with a Redis Pub/Sub or Kafka-backed implementation by implementing the `CacheEvictedEventBus` interface, though the Guava version suffices for single-node multi-instance testing.

## Multi-Instance Setup Example

The following Kotlin example demonstrates how to configure two distinct instances that share the same distributed cache and event bus, ensuring **cache consistency across multiple instances**:

```kotlin
import me.ahoo.cache.consistency.*
import me.ahoo.cache.api.client.ClientSideCache
import me.ahoo.cache.distributed.DistributedCache
import me.ahoo.cache.converter.KeyConverter

// Create shared infrastructure
val distributedCache: DistributedCache<String> = RedisCache()
val clientSideCache: ClientSideCache<String> = CaffeineCache()
val eventBus = GuavaCacheEvictedEventBus()
val factory = DefaultCoherentCacheFactory(eventBus)

val keyConverter = object : KeyConverter<Int> {
    override fun toStringKey(key: Int) = key.toString()
    override fun fromStringKey(stringKey: String) = stringKey.toInt()
}

// Instance A (JVM 1)
val cacheA = factory.create(
    CoherentCacheConfiguration(
        cacheName = "userCache",
        clientId = "instanceA",
        keyConverter = keyConverter,
        distributedCache = distributedCache,
        clientSideCache = clientSideCache
    )
)

// Instance B (JVM 2)
val cacheB = factory.create(
    CoherentCacheConfiguration(
        cacheName = "userCache",
        clientId = "instanceB",
        keyConverter = keyConverter,
        distributedCache = distributedCache,
        clientSideCache = clientSideCache
    )
)

// Write from Instance A
cacheA.setCache(42, CacheValue("Alice"))

// Read from Instance B
// L2 miss → L1 hit → updates B's L2 automatically
val value = cacheB.getCache(42) // Returns "Alice"

```

The test specification [`MultipleInstanceSyncSpec.kt`](https://github.com/ahoo-wang/cocache/blob/main/MultipleInstanceSyncSpec.kt) in the `cocache-test` module validates this behavior, simulating concurrent writes and reads across two cache instances to verify eventual consistency.

## Summary

- **CoCache uses a tiered architecture** combining distributed L1 (authoritative) and local L2 (fast) caches to balance performance with consistency.
- **Write operations** in [`DefaultCoherentCache.kt`](https://github.com/ahoo-wang/cocache/blob/main/DefaultCoherentCache.kt) update both layers and publish `CacheEvictedEvent` notifications via the event bus.
- **Remote instances** receive events through the `onEvicted()` handler and invalidate their local L2 entries, ensuring subsequent reads fetch fresh data from L1.
- **Cache stampede protection** is implemented via `keyLocks` in the `getCache()` method to prevent thundering herd scenarios during cache misses.
- **Eventual consistency** is achieved without distributed transactions, making the system suitable for high-throughput, low-latency applications.

## Frequently Asked Questions

### How does CoCache handle network partitions between instances?

CoCache guarantees **eventual consistency** rather than strong consistency. If the event bus becomes partitioned, instances will continue to serve stale data from their L2 caches until connectivity is restored and eviction events are delivered. The L1 distributed cache (Redis) remains authoritative, so reads will eventually converge to the correct values once the network heals or the L2 entries expire naturally.

### What happens if two instances update the same key simultaneously?

CoCache does not implement distributed locking for write conflicts. The last write to the L1 distributed cache wins based on the underlying storage’s consistency model (e.g., Redis single-threaded execution). Both instances will publish eviction events, causing all L2 caches to clear and reload the final value from L1 on the next access.

### Can I use a different message broker instead of Guava EventBus?

Yes. The `CacheEvictedEventBus` interface in `cocache-core` abstracts the messaging layer. You can implement this interface using Redis Pub/Sub, RabbitMQ, or Kafka by overriding the `publish()` and `register()` methods. The `DefaultCoherentCache` remains unchanged as long as the custom implementation delivers `CacheEvictedEvent` objects to all subscribed instances.

### Does CoCache support transactional updates across the cache and database?

CoCache does not provide built-in transaction coordination between the cache layers and external databases. When using `getCache()` with cache-aside loading, the data source write and cache update are separate operations. For atomicity, you should implement the **cache-aside** pattern at the application layer: update the database first, then call `setCache()` or `evict()` to invalidate the cache, accepting that a small window of inconsistency may exist between the two systems.