Moka Cache Lookup Performance: How the Lock-Free Design Achieves O(1) Speed

Moka delivers O(1) amortized lookup performance through a lock-free segmented hash map that uses atomic operations and crossbeam-epoch memory management, eliminating global locks and enabling sub-nanosecond read overhead compared to standard HashMap.

The moka-rs/moka crate is a high-performance caching library for Rust that prioritizes Moka cache lookup performance through innovative concurrency primitives. Unlike traditional caches that rely on coarse-grained locking, Moka implements a lock-free architecture that allows concurrent reads to scale linearly with CPU cores.

How Moka Cache Lookup Performance Works Under the Hood

The Lock-Free Segmented Hash Map Architecture

At the core of Moka's speed lies a lock-free, segmented hash map (cht::HashMap) implemented in src/cht/segment.rs. The map divides entries across multiple independent segments—defaulting to 2 × CPU count—with each segment owning an array of bucket pointers accessible via crossbeam_epoch::Atomic pointers.

This design eliminates global locks entirely. When a thread calls Cache::get, it hashes the key, selects a segment using high bits of the hash (segment_shift), and performs atomic reads on that segment's bucket array. The crossbeam-epoch memory management ensures safe, lock-free reclamation of old entries without stopping readers.

The Seven-Step Lookup Path

A single Cache::get call traverses a precise, optimized path through the codebase:

1. Public API Entry (src/sync/cache.rs, lines 995-1002): Cache::get forwards to BaseCache::get_with_hash.

2. Hash Computation (src/sync/base_cache.rs, lines 217-224): BaseCache::get_with_hash creates a closure to record the read operation and passes it to the internal routine.

3. Core Logic (src/sync/base_cache.rs, lines 265-272): BaseCache::do_get_with_hash checks whether the map is disabled, computes the current instant, and calls inner.get_key_value_and_then on the segmented hash map.

4. Segment Selection (src/cht/segment.rs, lines 282-288): The segmented hash map selects the appropriate segment using segment_shift on the hash, then reads the bucket array via an atomic pointer.

5. Expiration Check (src/sync/base_cache.rs, lines 297-304): The entry is validated against TTL/TTI policies and explicit invalidation flags. If expired, the entry is treated as a miss.

6. Metrics Recording (src/sync/base_cache.rs, lines 221-224): For valid entries, record(ReadOp::Get, now) updates the historic popularity estimator and refreshes the idle timer using atomic increments.

7. Return: The value is cloned or moved out and returned to the caller.

Notably, Cache::contains_key follows the same path but skips step 6, avoiding the atomic record operation and making it slightly faster for pure existence checks (src/sync/cache.rs, lines 671-678).

Optimizing Moka Cache Lookup Performance

While Moka's defaults provide excellent performance, you can tune specific levers for maximum throughput:

• Number of segments: Use CacheBuilder::num_segments(x) to increase shards beyond the default (2 × CPU count). More segments reduce contention under extreme thread counts at the cost of slightly higher memory usage.

• Hasher selection: Replace the default SipHash-1-3 with a faster non-cryptographic hasher like ahash using CacheBuilder::with_hasher(ahash::RandomState::new()). This significantly reduces hash computation time for small keys.

• Expiration policies: Disable TTL and TTI checks (time_to_live(None), time_to_idle(None)) if you need pure lookup speed. This skips the expiration validation step entirely.

• Read operation recording: Use CacheBuilder::record_read_ops(false) to disable the popularity estimator. This removes the atomic increment in step 6, yielding pure read performance comparable to a raw HashMap.

Code Examples for High-Performance Lookups

Here are practical implementations demonstrating optimized lookup patterns.

Synchronous High-Throughput Cache

use moka::sync::Cache;
use std::time::Duration;

fn main() {
    // Build a cache optimized for raw lookup speed
    let cache = Cache::builder()
        .max_capacity(1_000_000)
        .num_segments(8)               // More segments reduce contention
        .time_to_live(None)            // Disable expiration checks
        .record_read_ops(false)        // Skip popularity tracking
        .build();

    // Insert a value (expensive to compute only once)
    cache.insert("answer", 42);

    // O(1) lock-free lookup
    if let Some(v) = cache.get("answer") {
        println!("The answer is {v}");
    }

    // Existence check without affecting metrics
    let present = cache.contains_key("answer");
    println!("Key present: {present}");
}

Asynchronous Cache with Lazy Initialization

use moka::future::Cache;
use std::time::Duration;

#[tokio::main]
async fn main() {
    let cache = Cache::builder()
        .max_capacity(500_000)
        .time_to_idle(Some(Duration::from_secs(30)))
        .build();

    // get_with records the read and lazily creates if missing
    let val = cache
        .get_with("config", async {
            // Simulate async initialization
            tokio::time::sleep(Duration::from_millis(10)).await;
            "loaded_config".to_string()
        })
        .await;
    
    println!("config = {val}");
}

Key Source Files and Implementation Details

Understanding the lookup path requires familiarity with these specific files in the moka-rs/moka repository:

File	Role	Key Components
src/sync/cache.rs	Public API surface	Cache::get, Cache::contains_key, entry API
src/sync/base_cache.rs	Core lock-free logic	get_with_hash, do_get_with_hash, expiration checks
src/cht/segment.rs	Segmented hash map	Segment selection, segment_shift, bucket array access
src/cht/map/bucket.rs	Bucket storage	Linear probing, open addressing
src/common/concurrent/arc.rs	Reference counting	Lock-free Arc wrapper for entries
src/policy.rs	Cache policies	TTL, TTI, weight definitions affecting lookup validation

These files demonstrate how Moka transforms a simple get call into a lock-free segment lookup, expiration validation, and optional metrics recording—all executing in constant time with minimal CPU overhead.

Summary

• Moka achieves O(1) amortized lookup performance through a lock-free segmented hash map architecture that eliminates global locks.
• The lookup path traverses seven optimized steps: API entry, hash computation, core logic, segment selection, expiration checking, metrics recording, and return.
• Lock-free reads via crossbeam_epoch::Atomic pointers and segmented sharding (default 2× CPU count) enable linear scalability with thread count.
• You can optimize further by tuning segment count, switching to faster hashers like ahash, disabling expiration policies, and turning off read-op recording.

Frequently Asked Questions

What is the time complexity of Moka cache lookups?

Moka cache lookups run in O(1) amortized time complexity, similar to standard hash maps. Each lookup performs a constant amount of work: hashing the key, selecting a segment via bit shifting, following an atomic pointer to the bucket array, and validating the entry. The lock-free design ensures that thread contention does not degrade this complexity, maintaining constant-time performance even under high concurrency.

How does Moka achieve lock-free concurrency?

Moka implements lock-free concurrency using a segmented hash map built on crossbeam_epoch::Atomic pointers. Instead of locking the entire table, the map shards entries across multiple segments (defaulting to twice the CPU count). Each segment maintains its own bucket array accessible through atomic operations. The crossbeam-epoch memory management allows safe, lock-free reclamation of removed entries without stopping readers, eliminating global lock contention entirely.

Can I disable features to improve lookup speed?

Yes, you can disable several features to maximize Moka cache lookup performance. Use CacheBuilder::record_read_ops(false) to disable the popularity estimator and remove atomic increments from the read path. Set time_to_live(None) and time_to_idle(None) to skip expiration validation checks entirely. Additionally, increasing segments via num_segments() reduces contention, while switching to a faster hasher like ahash minimizes hash computation overhead.

How does Moka compare to standard HashMap performance?

Moka provides sub-nanosecond overhead compared to a raw std::collections::HashMap when configured optimally. The benchmark suite in the repository's benches/ directory demonstrates that Moka's lock-free reads, with features like read-op recording and expiration disabled, perform nearly identically to standard hash map lookups while adding thread safety and eviction policies. The primary difference emerges under high concurrency, where Moka's segmented design scales linearly while standard HashMap would require locking wrappers that degrade performance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}