How to Tune MaxInMemLogSize in Dragonboat to Prevent Memory Exhaustion

Set MaxInMemLogSize to at least 30% larger than your maximum proposal payload plus the internal Raft entry overhead (settings.EntryNonCmdFieldsSize, approximately 48 bytes) to bound memory usage while avoiding unnecessary ErrPayloadTooBig rejections.

The MaxInMemLogSize configuration parameter in the dragonboat Raft library controls how much unapplied log data a node buffers in memory. Without proper tuning, unbounded growth of the in-memory Raft log can lead to out-of-memory crashes, while overly conservative settings can throttle throughput with unnecessary proposal rejections.

What Is MaxInMemLogSize?

MaxInMemLogSize defines a soft limit on the total size of Raft log entries that have been accepted but not yet applied to the state machine. When a node receives a new proposal, it temporarily stores the entry in an in-memory buffer until the Raft state machine applies it and truncates the log. If the buffered size exceeds the configured target, the node rejects subsequent proposals to prevent unbounded memory growth.

How MaxInMemLogSize Prevents Memory Exhaustion

The limit is enforced by the payloadTooBig helper function in node.go, which checks whether accepting a new command would exceed the configured threshold:

// node.go – payload size check
func (n *node) payloadTooBig(sz int) bool {
    if n.config.MaxInMemLogSize == 0 {
        return false
    }
    return uint64(sz+settings.EntryNonCmdFieldsSize) > n.config.MaxInMemLogSize
}

When this function returns true, the proposal is rejected. Clients receive ErrPayloadTooBig when the specific command would exceed the remaining budget, or ErrSystemBusy when the node is already at capacity:

// request.go – error comment
// This might be caused when the Raft node reached its MaxInMemLogSize limit
var ErrSystemBusy = errors.New("system is too busy try again later")

Tuning Guidelines for MaxInMemLogSize

• Baseline Calculation: Set MaxInMemLogSize to much larger than the largest expected proposal payload plus the internal log entry overhead (settings.EntryNonCmdFieldsSize, approximately 48 bytes).
• Memory Budget: Compute the maximum total in-memory log size that your host can safely allocate (e.g., 10% of available RAM) and use that as the upper bound.
• Zero Value Behavior: Setting MaxInMemLogSize to 0 disables the limit (equivalent to math.MaxUint64). Use this only for testing; production workloads should set a concrete limit.
• Safety Margin: Add a safety margin of 20%–30% to the largest proposal size to accommodate multiple pending proposals that may coexist before being applied.
• Dynamic Workloads: If the workload has highly variable proposal sizes, err on the side of a larger limit and monitor ErrPayloadTooBig occurrences. Adjust downwards only after confirming the node never hits the limit.

Step-by-Step Configuration

1. Determine the maximum proposal size you will send (including any user payload).
2. Add Raft internal overhead – the constant settings.EntryNonCmdFieldsSize (approximately 48 bytes).
3. Apply a safety factor (e.g., multiply by 1.3).
4. Ensure the final number fits within your RAM budget (e.g., on an 8 GB machine, you may allow ≤ 800 MB for in-memory logs).
5. Assign the value to Config.MaxInMemLogSize when constructing the node configuration.

Code Examples

Setting MaxInMemLogSize in Node Configuration

import (
    "github.com/lni/dragonboat/v4"
    "github.com/lni/dragonboat/v4/config"
)

func startReplica() (*dragonboat.NodeHost, error) {
    cfg := config.Config{
        NodeID:       1,
        ElectionRTT:  10,
        HeartbeatRTT: 1,
        // Suppose the largest user payload is 2 MiB.
        // Add Raft overhead (settings.EntryNonCmdFieldsSize ≈ 48 bytes)
        // and a 30% safety margin.
        MaxInMemLogSize: uint64((2*1024*1024 + 48) * 1.3),
    }

    if err := cfg.Validate(); err != nil {
        return nil, err
    }

    nh, err := dragonboat.NewNodeHost(config.NodeHostConfig{})
    if err != nil {
        return nil, err
    }

    // Start replica with the tuned configuration
    err = nh.StartReplica(cfg, &myStateMachine{}, 0, 0)
    return nh, err
}

Handling ErrPayloadTooBig on the Client Side

import (
    "errors"
    "time"
    "github.com/lni/dragonboat/v4"
    "github.com/lni/dragonboat/v4/client"
)

func propose(session *client.Session, nh *dragonboat.NodeHost, data []byte) error {
    rs, err := nh.SyncPropose(session, data, 5*time.Second)
    if err != nil {
        if errors.Is(err, dragonboat.ErrPayloadTooBig) {
            // The in-memory log limit was reached.
            // Retry after letting the system apply pending entries, or
            // increase MaxInMemLogSize if memory permits.
            return err
        }
        return err
    }
    _ = rs
    return nil
}

Inspecting the Limit at Runtime

import (
    "fmt"
    "github.com/lni/dragonboat/v4"
)

func printLogLimit(node *dragonboat.Node) {
    cfg := node.GetConfig()
    fmt.Printf("Current MaxInMemLogSize = %d bytes\n", cfg.MaxInMemLogSize)
}

Key Source Files

File	Relevance	Link
config/config.go	Definition and validation of MaxInMemLogSize	config.go#L147-L157
node.go	Runtime check that enforces the limit (payloadTooBig)	node.go#L36-L41
request.go	Error comment indicating that MaxInMemLogSize may cause ErrSystemBusy	request.go#L64-L66

Summary

• MaxInMemLogSize acts as a soft limit on unapplied Raft log entries, rejecting new proposals via ErrPayloadTooBig when the buffer would exceed the configured size.
• The limit is enforced in node.go by the payloadTooBig helper, which adds settings.EntryNonCmdFieldsSize (≈48 bytes) to the payload size before comparison.
• Production deployments should set the value to at least 130% of the maximum expected payload plus overhead, while ensuring the total fits within 10–20% of available system memory.
• Setting MaxInMemLogSize to 0 disables protection entirely, which is only safe for testing.

Frequently Asked Questions

What happens when MaxInMemLogSize is exceeded?

When the in-memory log buffer approaches the limit, the payloadTooBig function in node.go returns true, causing the node to reject the proposal. The client receives dragonboat.ErrPayloadTooBig if the specific command would exceed the remaining budget, or dragonboat.ErrSystemBusy if the node is already at capacity. The application should either retry after a delay or increase the limit if memory permits.

Can I disable MaxInMemLogSize?

Setting MaxInMemLogSize to 0 effectively disables the limit by treating it as math.MaxUint64. This allows the in-memory log to grow unbounded until the process runs out of memory. This setting should be reserved for testing or development environments only; production workloads must specify a concrete limit to prevent OOM crashes.

How does MaxInMemLogSize relate to actual memory usage?

MaxInMemLogSize limits only the raw byte size of unapplied log entries stored in the Raft buffer. It does not account for Go runtime overhead, state machine memory, network buffers, or other node host structures. Therefore, the total process memory will be significantly higher than the configured limit. As a rule of thumb, ensure the host has at least 5–10× the MaxInMemLogSize available for the entire process.

What is EntryNonCmdFieldsSize and why does it matter?

settings.EntryNonCmdFieldsSize is a constant (approximately 48 bytes) representing the metadata overhead of a Raft log entry, including fields like the term and index. When payloadTooBig checks a proposal in node.go, it adds this overhead to the user payload size before comparing against MaxInMemLogSize. Failing to account for this overhead in your calculations can cause unexpected rejections of valid commands that appear to be under the limit.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}