Common Cassandra Performance Issues and Diagnostic Approaches: A Complete Troubleshooting Guide

High write latency, read spikes, CPU pressure, and compaction bottlenecks in Apache Cassandra typically stem from memtable limits, SSTable overlap, tombstone accumulation, or misconfigured thread pools—diagnosed via nodetool metrics, YAML configuration analysis, and source-code inspection of the compaction strategy.

Apache Cassandra is a distributed, write-optimized NoSQL database where performance depends on the efficient lifecycle of data from memtables to SSTables through compaction and reads. Understanding common Cassandra performance issues requires examining how write.flush logic, read amplification, and background compaction interact with JVM resources. This guide walks through the most frequent bottlenecks found in the apache/cassandra source code and provides systematic diagnostic approaches using built-in tooling and configuration analysis.

Common Cassandra Write Performance Issues

High Write Latency and Back-Pressure

Write stalls occur when memtable heap or off-heap limits are reached, forcing foreground writes to block until a flush completes. In cassandra.yaml, the memtable_cleanup_threshold and memtable_flush_writers settings control this behavior—insufficient writers cause large, infrequent flushes that create stop-the-world pauses.

According to the source documentation in src/java/org/apache/cassandra/db/memtable/Memtable_API.md, the flush trigger logic monitors the MemtablePool allocation. When the pool exceeds its configured limit, the Memtable.flush operation must complete before new writes proceed, creating observable back-pressure in application metrics.

Common Cassandra Read Performance Issues

Read Latency Spikes and SSTable Overlap

Read amplification occurs when queries must scan multiple overlapping SSTables to reconstruct a partition. The maxOverlap metric—calculated in src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.java—indicates how many files must be read per partition key.

Sub-optimal compaction leaves many overlapping SSTables on disk, while insufficient concurrent_reads in cassandra.yaml creates disk I/O contention. Additionally, missing or undersized caches (key cache, row cache, index-summary) force full disk reads for frequently accessed data.

Tombstone Pressure

Excessive tombstones from DELETE operations or expired TTLs accumulate in SSTables and must be scanned during reads and compactions. The test utilities in src/java/org/apache/cassandra/utils/CassandraGenerators.java demonstrate how tombstone-heavy workloads create GC pressure and slow queries.

Monitor Average tombstones per slice via nodetool cfstats—values exceeding 1% of row counts indicate severe performance degradation during range scans.

Compaction Bottlenecks

Throughput Limitations

Compaction bottlenecks manifest when concurrent_compactors (defined in cassandra.yaml) is too low for the data volume. The UnifiedCompactionStrategy uses controller.getNumShards and fan-factor logic to determine parallel compaction jobs—misconfiguration here leads to large compaction tasks that saturate I/O.

In UnifiedCompactionStrategy.java, the getMaximalTasks method selects compaction picks based on overlap calculations. When the strategy detects high overlap but cannot execute compactions due to thread starvation, read performance degrades exponentially.

Index-Related Slowdowns

SASI (SStable Attached Secondary Index) and secondary indexes on high-cardinality columns create large index files that must be scanned per query. As documented in doc/SASI.md, these indexes perform poorly with ALLOW FILTERING queries that force full table scans across multiple SSTables.

Diagnostic Approaches for Cassandra Performance Issues

Step 1: Collect Node Metrics

Use nodetool commands to establish baseline performance:

• nodetool cfstats: Exposes per-table read/write latency, SSTable counts, and tombstone averages
• nodetool compactionstats: Shows pending compactions and active compaction throughput
• nodetool tpstats: Reveals thread pool utilization for ReadStage and MutationStage

Step 2: Inspect Configuration

Verify critical settings in cassandra.yaml:

• concurrent_reads: Should equal 16 × number of spindles (or SSDs)
• concurrent_writes: Recommended at 8 × number of cores
• concurrent_compactors: Set to 2 × number of disks minimum
• memtable_heap_space_in_mb and memtable_offheap_space_in_mb: Should total roughly ¼ of heap size

Step 3: Analyze SSTable Overlap

Check level distribution and overlap using:

nodetool upgradesstables -dryrun
sstablemetadata <sstable-file>

High overlap indicates compaction lag requiring intervention.

Step 4: Review Compaction Logs

Examine system.log for UnifiedCompactionStrategy trace entries. The source code contains log.trace statements indicating when compaction picks are made, the overlap count selected, and whether the pick was capped due to resource constraints.

Step 5: Profile JVM Resources

For CPU or G1 GC pressure, attach a JDK profiler (e.g., jvisualvm or async-profiler) and focus sampling on:

• Memtable.flush operations
• CompactionTask.run execution
• Garbage collection cycles during heavy write workloads

Configuration Tuning Checklist

Area	Configuration	Verification Method
Concurrent Reads	16 × number_of_spindles in cassandra.yaml	nodetool info → Read Thread Pool < 70% utilization
Concurrent Writes	8 × number_of_cores in cassandra.yaml	nodetool tpstats → MutationStage queue length near zero
Compaction Threads	2 × number_of_disks via concurrent_compactors	nodetool compactionstats showing near-zero pending tasks
Memtable Sizing	¼ of heap for heap + off-heap combined	MemtablePool metrics showing flush latency < 500ms
Cache Configuration	Key cache ≈ 5% heap, index-summary ≈ 5% heap	nodetool info showing > 80% hit rates
Tombstone Management	Appropriate gc_grace_seconds and TTL values	nodetool cfstats → Average tombstones per slice < 1% of rows

Code Examples for Diagnosing Cassandra Performance

Java: Inspect Compaction Overlap Programmatically

The following snippet accesses the same compaction selection logic used by Cassandra's background threads:

// Obtain the UnifiedCompactionStrategy for a table
ColumnFamilyStore cfs = Schema.instance.getTableMetadata("myks", "mytable")
                                   .getColumnFamilyStore();
UnifiedCompactionStrategy ucs = (UnifiedCompactionStrategy) cfs.getCompactionStrategyManager()
                                                             .getCustomStrategy();
long now = System.currentTimeMillis();
List<AbstractCompactionTask> tasks = ucs.getMaximalTasks(now, false);
System.out.println("Selected compaction overlap: " + 
    tasks.stream().mapToInt(t -> ((UnifiedCompactionStrategy.CompactionPick) t).overlap).max().orElse(0));

This reveals the current maxOverlap value driving read amplification for specific tables.

CQL: Identify Hot Partitions

When read latency spikes correlate with specific data access patterns:

SELECT token(partition_key), COUNT(*) AS rows
FROM myks.mytable
GROUP BY token(partition_key)
ORDER BY rows DESC
LIMIT 10;

Large token ranges with concentrated row counts indicate partition hotspots causing SSTable read amplification.

Bash: Force Targeted Compaction

For immediate remediation of SSTable overlap:

nodetool compact myks mytable

# Limit to specific shard count (UCS specific):

nodetool compact -j 4 myks mytable

The -j flag maps to the concurrent_compactors-controlled shard count used inside UnifiedCompactionStrategy.createCompactionTasks where numShards is calculated.

Key Source Files for Performance Analysis

File	Performance Relevance
src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.java	Contains the read-amplification calculation logic and fan-factor selection for compaction picks
conf/cassandra.yaml	Central configuration for thread pools, caches, memtables, and disk settings
src/java/org/apache/cassandra/db/memtable/Memtable_API.md	Documents memtable flushing thresholds and allocation types affecting write stalls
doc/SASI.md	Performance characteristics and limitations of SASI indexing
test/unit/org/apache/cassandra/utils/CassandraGenerators.java	Reference for tombstone generation patterns and test simulations

Summary

• Write latency issues typically indicate memtable pressure or insufficient flush writers—monitor MemtablePool metrics and adjust cassandra.yaml thresholds.
• Read latency spikes usually result from SSTable overlap or tombstone accumulation—check UnifiedCompactionStrategy overlap calculations and cfstats tombstone counts.
• CPU and GC pressure often stem from large heap allocations or excessive tombstone scanning—size caches appropriately and enforce TTL limits.
• Compaction bottlenecks require balancing concurrent_compactors against disk I/O capacity using UCS fan-factor settings.
• Index slowdowns occur with SASI on high-cardinality columns or ALLOW FILTERING queries—avoid these patterns or use alternative indexing strategies.

Frequently Asked Questions

What causes sudden write latency increases in Cassandra?

Sudden write latency spikes typically occur when memtable limits are exceeded and writes stall waiting for flush operations to complete. Check nodetool tpstats for blocked MutationStage tasks and verify memtable_cleanup_threshold settings in cassandra.yaml allow sufficient off-heap space for your write throughput.

How do I identify if compaction is causing read performance issues?

Run nodetool compactionstats to check for pending compactions, then use sstablemetadata to examine SSTable overlap levels. High overlap counts indicate that UnifiedCompactionStrategy is unable to keep up with write volume, forcing reads to scan multiple files per partition. Increasing concurrent_compactors or adjusting the fan-factor may resolve this.

What is the recommended approach for diagnosing hot partitions?

Use CQL to group rows by partition token and identify skewed distributions, then correlate these with nodetool cfstats output showing high read latency on specific tables. Hot partitions often appear as tokens with disproportionately high row counts relative to the cluster average, causing uneven SSTable access patterns.

When should I be concerned about tombstone performance impact?

Concern is warranted when nodetool cfstats reports Average tombstones per slice exceeding 1% of your average row count per partition. High tombstone counts force Cassandra to scan and filter deleted records during reads, increasing latency and GC pressure. Remediate by ensuring gc_grace_seconds is appropriately set and running nodetool compact after bulk deletions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}