Implementing Size-Aware Eviction in Moka: A Complete Guide to Weight-Based Caching

Moka supports size-aware eviction by accepting a user-defined weigher closure that assigns a weight to each entry, then tracking the total weighted size in atomic counters to enforce byte-based capacity limits alongside count-based eviction.

The moka-rs/moka crate provides a high-performance concurrent cache for Rust that goes beyond simple entry counting. By implementing size-aware eviction in Moka, you can configure the cache to limit its total memory footprint in bytes rather than just the number of items. This is essential for applications storing variable-sized blobs, images, or serialized data where a count limit does not correlate with actual memory pressure.

Configuring the Weigher Closure

Size-aware eviction begins with the weigher, a user-provided closure that calculates the weight of each key-value pair at insertion time.

The CacheBuilder API

The CacheBuilder::weigher method in src/sync/builder.rs (lines 414–424) stores the closure in the builder state:

pub fn weigher(self, weigher: impl Fn(&K, &V) -> u32 + Send + Sync + 'static) -> Self {
    Self {
        weigher: Some(Arc::new(weigher)),
        ..self
    }
}

This closure must return a u32 representing the weight, typically measured in bytes, and must be thread-safe (Send + Sync + 'static).

Storage in the Cache Core

During CacheBuilder::build, the weigher is passed through BaseCache::new to Inner::new in src/sync/base_cache.rs (lines 140–149):

let inner = Arc::new(Inner::new(
    …,
    weigher,
    …,
));

Here it becomes the field weigher: Option<Weigher<K, V>> inside the cache's internal state, making it available to all insert and update operations.

Calculating Entry Weights at Runtime

When entries are inserted or updated, Moka evaluates their weight before admitting them to the cache.

Weight Resolution During Insertion

In BaseCache::do_insert_with_hash (src/sync/base_cache.rs, lines 488–490), the cache queries the weight for the candidate entry:

let weight = self.inner.weigh(&key, &value);

Default Weight Fallback

The Inner::weigh method (src/sync/base_cache.rs, lines 440–442) invokes the stored closure if present; otherwise, it assigns a default weight of 1, which effectively enables count-based eviction:

fn weigh(&self, key: &K, value: &V) -> u32 {
    self.weigher.as_ref().map_or(1, |w| w(key, value))
}

This returned weight is stored in the entry's EntryInfo as policy_weight and used to update global counters.

Tracking Global Weighted Size

Moka maintains two atomic counters in the Inner struct to track cache state:

• entry_count: AtomicCell<u64> — total number of entries
• weighted_size: AtomicCell<u64> — sum of all entry weights

Updating Counters on Upsert and Eviction

During BaseCache::handle_upsert (src/sync/base_cache.rs, lines 614–629), these counters are updated using saturating arithmetic to prevent overflow:

counters.saturating_add(0, new_weight);   // on insert / update
counters.saturating_sub(0, old_weight);   // on replace / removal

When an entry is evicted or replaced, its previous weight is subtracted from weighted_size, ensuring the global metric remains accurate.

Enforcing Byte-Based Capacity Limits

The eviction logic uses the weighted_size counter to enforce the max_capacity specified in the builder.

Admission Control

Before admitting a new entry, has_enough_capacity (src/sync/base_cache.rs, lines 220–226) verifies that the candidate fits within the limit:

fn has_enough_capacity(&self, candidate_weight: u32, counters: &EvictionCounters) -> bool {
    self.max_capacity.map_or(true, |limit| {
        counters.weighted_size + candidate_weight as u64 <= limit
    })
}

If the candidate's weight exceeds the entire max_capacity, it is rejected immediately and a RemovalCause::Size eviction is reported.

Target Weight Calculation for Eviction

When the cache exceeds its limit, weights_to_evict (src/sync/base_cache.rs, lines 262–268) calculates the deficit:

fn weights_to_evict(&self, counters: &EvictionCounters) -> u64 {
    self.max_capacity
        .map(|limit| counters.weighted_size.saturating_sub(limit))
        .unwrap_or_default()
}

The eviction loop (Inner::do_run_pending_tasks) then calls evict_lru_entries with this target weight, removing the least-recently-used entries until the required bytes are freed.

Integration with Eviction Policies

Moka's TinyLFU and LRU policies operate transparently with weighted entries. The admission decision receives a candidate struct that already carries the policy_weight, allowing the eviction algorithms to treat weight as a numeric attribute without special casing. This design ensures that frequency-based and recency-based eviction remain efficient even when entries vary dramatically in size.

Complete Implementation Example

The following example demonstrates a size-aware cache that limits total storage to 32 MiB, weighing each String by its UTF-8 byte length:

use moka::sync::Cache;

let cache = Cache::builder()
    // Each value's weight is its byte length.
    .weigher(|_key, value: &String| value.len().try_into().unwrap_or(u32::MAX))
    // Allow up to 32 MiB of total weight.
    .max_capacity(32 * 1024 * 1024)
    .build();

cache.insert(0, "short".to_string());                     // weight = 5
cache.insert(1, "a".repeat(10 * 1024 * 1024));           // weight ≈ 10 MiB
// Inserting a massive entry will trigger eviction of the least-recently-used
// entries until the total weight is ≤ 32 MiB.
cache.insert(2, "b".repeat(30 * 1024 * 1024));

This pattern is available in the repository at examples/size_aware_eviction_sync.rs.

Async Cache Support

The same size-aware eviction API is available for asynchronous caches via moka::future::Cache. The builder logic in src/future/builder.rs mirrors the synchronous implementation, allowing you to use .weigher() and .max_capacity() identically in async contexts without blocking the runtime.

Summary

• Weigher closure: Define custom weight logic via CacheBuilder::weigher in src/sync/builder.rs, stored as Option<Weigher<K, V>> in the cache core.
• Weight calculation: Performed at insertion time in BaseCache::do_insert_with_hash, defaulting to 1 for count-based eviction.
• Global tracking: Atomic counters weighted_size and entry_count in Inner maintain real-time totals, updated via saturating arithmetic in handle_upsert.
• Capacity enforcement: has_enough_capacity rejects oversized entries, while weights_to_evict drives the LRU eviction loop to free specific byte targets.
• Policy integration: TinyLFU and LRU policies handle weights as standard numeric attributes, requiring no special configuration.

Frequently Asked Questions

What happens if a single entry's weight exceeds max_capacity?

The entry is rejected before insertion. The has_enough_capacity check in src/sync/base_cache.rs compares the candidate weight against the limit; if larger, it reports a RemovalCause::Size and drops the entry.

When is the weigher closure invoked?

The weigher is called during write operations—specifically in BaseCache::do_insert_with_hash when inserting or updating an entry. It is not invoked on read operations, ensuring that lookups remain fast and do not execute arbitrary user code.

Can I change the weigher after building the cache?

No. The weigher is stored as an Option<Arc<dyn Fn(...)>> inside the Inner struct at construction time and cannot be modified afterward. To use different weighting logic, you must create a new Cache instance with a different builder configuration.

Does size-aware eviction work with the async Cache?

Yes. The moka::future::Cache type supports identical size-aware eviction via src/future/builder.rs, which provides the same weigher and max_capacity methods as the synchronous builder.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}