How CoCache's Multi-Level Caching Architecture Works: L1 Client-Side vs L2 Distributed

CoCache implements a two-level caching hierarchy where L1 serves as a fast, client-side in-process cache and L2 acts as a shared distributed store, coordinated by DefaultCoherentCache to ensure consistency through event-driven eviction propagation.

The ahoo-wang/cocache library provides a coherent caching solution that bridges local JVM performance with distributed consistency. By combining an L1 client-side cache (in-process) with an L2 distributed cache (Redis, Memcached), CoCache minimizes network latency for hot data while maintaining cross-instance coherence through a publish-subscribe eviction mechanism.

Architecture Overview

CoCache's multi-level design centers on the DefaultCoherentCache class, which orchestrates data flow between three tiers: L0 (the underlying data source), L1 (local JVM cache), and L2 (distributed shared cache). The architecture is configured through CoherentCacheConfiguration, allowing developers to plug in specific implementations for each level.

The L1 cache defaults to MapClientSideCache, a simple in-process ConcurrentHashMap stored within the application heap. The L2 cache implements the DistributedCache interface, typically backed by Redis or similar stores. This separation allows microservices to serve repeated requests from local memory while ensuring that cache updates propagate to all nodes via the distributed layer.

Configuration and Setup

Cache hierarchy behavior is defined in CoherentCacheConfiguration.kt. The data class accepts both cache implementations along with consistency settings:

data class CoherentCacheConfiguration<K, V>(
    override val cacheName: String,
    override val clientId: String,
    val keyConverter: KeyConverter<K>,
    val distributedCache: DistributedCache<V>,        // L2
    val clientSideCache: ClientSideCache<V> = MapClientSideCache(),  // L1
    val cacheSource: CacheSource<K, V> = CacheSource.noOp(),
    val keyFilter: KeyFilter = NoOpKeyFilter
)

To instantiate a coherent cache with Redis as L2 and the default in-process L1:

val redisCache: DistributedCache<String> = RedisCacheBuilder
    .builder()
    .host("localhost")
    .port(6379)
    .build()

val config = CoherentCacheConfiguration(
    cacheName = "userCache",
    clientId = UUID.randomUUID().toString(),
    keyConverter = StringKeyConverter(),
    distributedCache = redisCache,
    clientSideCache = MapClientSideCache()
)

val cache = DefaultCoherentCacheFactory().create(config)

Read Path and Cache Lookup Flow

The getCache method in DefaultCoherentCache.kt implements a hierarchical lookup strategy with cache breakdown protection. As implemented in lines 49-75, the method follows a strict L1 → L2 → L0 progression:

override fun getCache(key: K): CacheValue<V>? {
    val cacheKey = keyConverter.toStringKey(key)
    
    // Step 1: Attempt L1 (client-side) lookup
    getL2Cache(cacheKey)?.let { return it }
    
    // Step 2: Acquire fine-grained lock to prevent thundering herd
    val lock = getLock(cacheKey)
    synchronized(lock) {
        // Double-check after lock acquisition
        getL2Cache(cacheKey)?.let { return it }
        
        // Step 3: Load from L0 (source) and populate both levels
        cacheSource.loadCacheValue(key)?.let {
            setCache(cacheKey, it)
            return it
        }
        
        // Step 4: Store missing guard for negative caching
        setCache(cacheKey, DefaultCacheValue.missingGuard(ttl, ttlAmplitude))
        return null
    }
}

L1 promotion occurs automatically when L2 returns a hit. The getL2Cache helper method first consults clientSideCache.getCache(). If absent, it queries distributedCache.getCache() and, upon success, writes the value back to L1 via clientSideCache.setCache(). This ensures subsequent requests for the same key resolve entirely in local memory.

Write Path and Consistency Guarantees

Writes propagate atomically to both levels. The setCache implementation (lines 42-48 in DefaultCoherentCache.kt) updates L1 and L2 simultaneously before broadcasting an eviction event:

override fun setCache(key: K, value: CacheValue<V>) {
    if (value.isExpired) return
    val cacheKey = keyConverter.toStringKey(key)
    clientSideCache.setCache(cacheKey, value)    // L1 write
    distributedCache.setCache(cacheKey, value)   // L2 write
    cacheEvictedEventBus.publish(
        CacheEvictedEvent(cacheName, cacheKey, clientId)
    )
}

This write-through pattern ensures that the distributed layer always holds the latest value, while the accompanying CacheEvictedEvent signals other application instances to invalidate their local L1 copies. The clientId field prevents nodes from processing their own eviction events.

Eviction Propagation and Coherence

Cache coherence relies on an event-driven invalidation strategy. When any node evicts a key, the CacheEvictedEventBus distributes the event to all registered listeners. The onEvicted handler (lines 58-80 in DefaultCoherentCache.kt) processes these messages:

@Subscribe
override fun onEvicted(cacheEvictedEvent: CacheEvictedEvent) {
    if (cacheEvictedEvent.cacheName != cacheName) return
    if (cacheEvictedEvent.publisherId == clientId) return  // Ignore self
    
    // Evict only from L1; L2 is already updated by the publisher
    clientSideCache.evict(cacheEvictedEvent.key)
}

This approach maintains eventual consistency across the cluster. The originating node updates L2, while peer nodes purge their L1 snapshots upon receiving the event, forcing them to fetch fresh data from L2 on the next access.

Concurrency Control and Cache Breakdown Protection

CoCache prevents thundering-herd scenarios through fine-grained locking per cache key. The keyLocks map stores key-specific lock objects, ensuring that only one thread executes the L0 data source query when both cache levels miss:

private fun getLock(key: String): Any {
    return keyLocks.computeIfAbsent(key) { Any() }
}

private fun releaseLock(key: String) {
    keyLocks.remove(key)
}

Locks are created on demand via computeIfAbsent and released after the critical section completes. This mechanism, defined in lines 77-86 of DefaultCoherentCache.kt, guarantees that expensive database queries execute exactly once per cache key during a miss storm.

Summary

• Two-level hierarchy: L1 (ClientSideCache) provides sub-millisecond local access, while L2 (DistributedCache) ensures cross-node consistency.
• Hierarchical reads: Lookups cascade from L1 → L2 → L0, with automatic promotion from distributed to local cache.
• Atomic writes: The setCache method updates both levels and publishes eviction events to maintain coherence.
• Event-driven eviction: CacheEvictedEvent bus ensures that L1 entries are invalidated cluster-wide when data changes.
• Thundering-herd protection: Per-key locks in DefaultCoherentCache serialize access to the underlying data source during cache misses.

Frequently Asked Questions

What happens when L1 and L2 both miss?

When both cache levels miss, CoCache acquires a per-key lock and queries the L0 source (database or remote service). The result is written to both L1 and L2 before releasing the lock, ensuring subsequent requests hit the cache. If the source returns null, a "missing guard" placeholder is stored to prevent repeated useless queries.

How does CoCache handle cache eviction across multiple application instances?

Eviction uses a pub-sub model. When one instance writes or evicts a key, it publishes a CacheEvictedEvent to the bus. Other instances receive this event and evict the key from their local L1 only, leaving L2 (the shared source of truth) intact. This ensures all nodes refresh their local snapshots on the next read.

Can I disable the L1 client-side cache?

Yes. While CoherentCacheConfiguration defaults to MapClientSideCache(), you can pass a no-op implementation of ClientSideCache or configure the cache to use L2 exclusively. However, this eliminates the performance benefits of local heap access and increases load on the distributed cache.

What distributed cache implementations does CoCache support?

The DistributedCache interface in cocache-api supports any backing store. The repository provides concrete implementations for Redis (via cocache-spring-redis) and can be extended for Memcached, Hazelcast, or other systems. The L2 implementation is injected into CoherentCacheConfiguration at instantiation time.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}