How to Handle Node Decommissioning with transfer_hints_on_decommission in Apache Cassandra

When decommissioning a Cassandra node, the transfer_hints_on_decommission configuration flag determines whether the system streams pending hints to remaining replicas or permanently deletes them, directly trading data durability for decommission speed.

Apache Cassandra's node decommissioning process involves the UnbootstrapStreams sequence, which manages the transition of data ownership when a node leaves the cluster. The transfer_hints_on_decommission parameter controls how the system handles write hints that were stored for the decommissioning node, ensuring administrators can balance between complete data preservation and rapid cluster reconfiguration.

Understanding transfer_hints_on_decommission in Cassandra

The Role of Hints During Decommission

When a node prepares to leave the cluster, other nodes may have stored hints—temporary write buffers—for that departing node. The transfer_hints_on_decommission boolean flag, defined in cassandra.yaml, dictates whether these hints are preserved or discarded during the decommission workflow.

Source Code Implementation

According to the Apache Cassandra source code, the core logic resides in src/java/org/apache/cassandra/tcm/sequences/UnbootstrapStreams.java. During the unbootstrap sequence, the system checks DatabaseDescriptor.getTransferHintsOnDecommission() to determine the appropriate hint handling strategy after batch-log replay completes.

How Hint Transfer Works During Node Decommission

The decommissioning process follows a specific execution path based on the flag's value:

If enabled (true): The system invokes StorageService.instance.streamHints() to transfer all pending hints to the remaining replica nodes. This ensures no write data is lost but increases network traffic and extends decommission duration.

If disabled (false): Cassandra immediately pauses the HintsService, stops hint dispatch, and deletes all pending hints on the leaving node. This accelerates decommission completion but permanently discards any writes stored exclusively as hints.

Configuring transfer_hints_on_decommission

You can configure this behavior through static YAML files or dynamic JMX operations without restarting the node.

Static Configuration in cassandra.yaml

Add or modify the following line in your cassandra.yaml configuration file:

# cassandra.yaml

transfer_hints_on_decommission: true

Set to false to disable hint transfer and speed up decommissioning at the risk of data loss.

Dynamic Configuration via JMX

Change the setting at runtime using JMX to adapt to changing cluster conditions without service interruption:

import org.apache.cassandra.service.StorageServiceMBean;
import javax.management.MBeanServerConnection;
import javax.management.ObjectName;

// Connect to MBeanServerConnection mbsc
ObjectName ssName = new ObjectName("org.apache.cassandra.db:type=StorageService");
StorageServiceMBean ss = javax.management.JMX.newMBeanProxy(mbsc, ssName, StorageServiceMBean.class);

// Enable hint transfer dynamically
ss.setTransferHintsOnDecommission(true);

The JMX interface is defined in src/java/org/apache/cassandra/service/StorageServiceMBean.java, with the concrete implementation in src/java/org/apache/cassandra/service/StorageService.java delegating to DatabaseDescriptor.

Executing Node Decommission with Hint Handling

Once configured, initiate decommissioning using nodetool:

# Standard decommission with hint transfer based on configuration

nodetool decommission

# Force decommission even if it reduces replication factor

nodetool decommission --force

During execution, the UnbootstrapStreams sequence performs the following operations:

1. Replays the batch-log to ensure data consistency.
2. Checks DatabaseDescriptor.getTransferHintsOnDecommission() at lines 33-38 of UnbootstrapStreams.java.
3. Either streams hints via StorageService.instance.streamHints() or pauses and deletes them based on the flag value.

Performance Implications and Best Practices

Choosing the appropriate setting depends on your data durability requirements and network capacity:

• Enable transfer when running critical workloads where hint data must not be lost. Expect increased decommission time proportional to hint volume and network bandwidth.
• Disable transfer in development environments or when the cluster can tolerate potential write loss. This minimizes decommission duration and reduces network congestion.

The flag can be toggled at runtime, allowing you to test behavior in staging environments or adjust for specific maintenance windows without cluster restarts.

Summary

• The transfer_hints_on_decommission flag in cassandra.yaml controls whether hints are preserved or deleted during node decommissioning.
• When enabled, StorageService.instance.streamHints() transfers pending hints to remaining replicas via the UnbootstrapStreams sequence.
• When disabled, Cassandra pauses HintsService and permanently deletes all hints on the leaving node.
• Configuration is available statically in YAML or dynamically via JMX through StorageServiceMBean.
• Source code implementation spans UnbootstrapStreams.java, DatabaseDescriptor.java, and StorageService.java.

Frequently Asked Questions

What happens to hints when transfer_hints_on_decommission is set to false?

When disabled, Cassandra immediately pauses the HintsService, stops all hint dispatch operations, and permanently deletes every pending hint stored on the decommissioning node. Any writes that were only stored as hints are lost, though the decommission process completes faster.

Can I change transfer_hints_on_decommission without restarting Cassandra?

Yes. You can modify this setting dynamically at runtime using JMX via the StorageServiceMBean interface. Call setTransferHintsOnDecommission(true) or false through your JMX client to adjust behavior without restarting the node.

Where is the transfer_hints_on_decommission logic implemented in the source code?

The decision logic resides in src/java/org/apache/cassandra/tcm/sequences/UnbootstrapStreams.java at lines 33-38. The configuration getter and setter are implemented in src/java/org/apache/cassandra/config/DatabaseDescriptor.java, while JMX exposure occurs in src/java/org/apache/cassandra/service/StorageServiceMBean.java and StorageService.java.

Does enabling hint transfer significantly slow down decommissioning?

Yes. Enabling transfer_hints_on_decommission adds network overhead because the system must stream all pending hints to remaining replicas using StorageService.instance.streamHints(). The duration increases with hint volume and available bandwidth, but ensures no write data is lost during the node removal process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}