# How to Tune MaxInMemLogSize in Dragonboat to Prevent Memory Exhaustion

> Effectively tune MaxInMemLogSize in Dragonboat to prevent memory exhaustion. Learn essential settings to bound memory usage and avoid errors.

- Repository: [lni/dragonboat](https://github.com/lni/dragonboat)
- Tags: performance
- Published: 2026-03-06

---

**Set `MaxInMemLogSize` to at least 30% larger than your maximum proposal payload plus the internal Raft entry overhead (`settings.EntryNonCmdFieldsSize`, approximately 48 bytes) to bound memory usage while avoiding unnecessary `ErrPayloadTooBig` rejections.**

The `MaxInMemLogSize` configuration parameter in the [dragonboat](https://github.com/lni/dragonboat) Raft library controls how much unapplied log data a node buffers in memory. Without proper tuning, unbounded growth of the in-memory Raft log can lead to out-of-memory crashes, while overly conservative settings can throttle throughput with unnecessary proposal rejections.

## What Is MaxInMemLogSize?

`MaxInMemLogSize` defines a **soft limit** on the total size of Raft log entries that have been accepted but not yet applied to the state machine. When a node receives a new proposal, it temporarily stores the entry in an in-memory buffer until the Raft state machine applies it and truncates the log. If the buffered size exceeds the configured target, the node rejects subsequent proposals to prevent unbounded memory growth.

## How MaxInMemLogSize Prevents Memory Exhaustion

The limit is enforced by the `payloadTooBig` helper function in [`node.go`](https://github.com/lni/dragonboat/blob/main/node.go), which checks whether accepting a new command would exceed the configured threshold:

```go
// node.go – payload size check
func (n *node) payloadTooBig(sz int) bool {
    if n.config.MaxInMemLogSize == 0 {
        return false
    }
    return uint64(sz+settings.EntryNonCmdFieldsSize) > n.config.MaxInMemLogSize
}

```

When this function returns `true`, the proposal is rejected. Clients receive `ErrPayloadTooBig` when the specific command would exceed the remaining budget, or `ErrSystemBusy` when the node is already at capacity:

```go
// request.go – error comment
// This might be caused when the Raft node reached its MaxInMemLogSize limit
var ErrSystemBusy = errors.New("system is too busy try again later")

```

## Tuning Guidelines for MaxInMemLogSize

- **Baseline Calculation**: Set `MaxInMemLogSize` to **much larger** than the largest expected proposal payload plus the internal log entry overhead (`settings.EntryNonCmdFieldsSize`, approximately 48 bytes).
- **Memory Budget**: Compute the maximum total in-memory log size that your host can safely allocate (e.g., 10% of available RAM) and use that as the upper bound.
- **Zero Value Behavior**: Setting `MaxInMemLogSize` to `0` disables the limit (equivalent to `math.MaxUint64`). Use this only for testing; production workloads should set a concrete limit.
- **Safety Margin**: Add a safety margin of 20%–30% to the largest proposal size to accommodate multiple pending proposals that may coexist before being applied.
- **Dynamic Workloads**: If the workload has highly variable proposal sizes, err on the side of a larger limit and monitor `ErrPayloadTooBig` occurrences. Adjust downwards only after confirming the node never hits the limit.

## Step-by-Step Configuration

1. **Determine the maximum proposal size** you will send (including any user payload).
2. **Add Raft internal overhead** – the constant `settings.EntryNonCmdFieldsSize` (approximately 48 bytes).
3. **Apply a safety factor** (e.g., multiply by 1.3).
4. **Ensure the final number fits within your RAM budget** (e.g., on an 8 GB machine, you may allow ≤ 800 MB for in-memory logs).
5. **Assign the value to `Config.MaxInMemLogSize`** when constructing the node configuration.

## Code Examples

### Setting MaxInMemLogSize in Node Configuration

```go
import (
    "github.com/lni/dragonboat/v4"
    "github.com/lni/dragonboat/v4/config"
)

func startReplica() (*dragonboat.NodeHost, error) {
    cfg := config.Config{
        NodeID:       1,
        ElectionRTT:  10,
        HeartbeatRTT: 1,
        // Suppose the largest user payload is 2 MiB.
        // Add Raft overhead (settings.EntryNonCmdFieldsSize ≈ 48 bytes)
        // and a 30% safety margin.
        MaxInMemLogSize: uint64((2*1024*1024 + 48) * 1.3),
    }

    if err := cfg.Validate(); err != nil {
        return nil, err
    }

    nh, err := dragonboat.NewNodeHost(config.NodeHostConfig{})
    if err != nil {
        return nil, err
    }

    // Start replica with the tuned configuration
    err = nh.StartReplica(cfg, &myStateMachine{}, 0, 0)
    return nh, err
}

```

### Handling ErrPayloadTooBig on the Client Side

```go
import (
    "errors"
    "time"
    "github.com/lni/dragonboat/v4"
    "github.com/lni/dragonboat/v4/client"
)

func propose(session *client.Session, nh *dragonboat.NodeHost, data []byte) error {
    rs, err := nh.SyncPropose(session, data, 5*time.Second)
    if err != nil {
        if errors.Is(err, dragonboat.ErrPayloadTooBig) {
            // The in-memory log limit was reached.
            // Retry after letting the system apply pending entries, or
            // increase MaxInMemLogSize if memory permits.
            return err
        }
        return err
    }
    _ = rs
    return nil
}

```

### Inspecting the Limit at Runtime

```go
import (
    "fmt"
    "github.com/lni/dragonboat/v4"
)

func printLogLimit(node *dragonboat.Node) {
    cfg := node.GetConfig()
    fmt.Printf("Current MaxInMemLogSize = %d bytes\n", cfg.MaxInMemLogSize)
}

```

## Key Source Files

| File | Relevance | Link |
|------|-----------|------|
| [`config/config.go`](https://github.com/lni/dragonboat/blob/main/config/config.go) | Definition and validation of `MaxInMemLogSize` | [config.go#L147-L157](https://github.com/lni/dragonboat/blob/master/config/config.go#L147-L157) |
| [`node.go`](https://github.com/lni/dragonboat/blob/main/node.go) | Runtime check that enforces the limit (`payloadTooBig`) | [node.go#L36-L41](https://github.com/lni/dragonboat/blob/master/node.go#L36-L41) |
| [`request.go`](https://github.com/lni/dragonboat/blob/main/request.go) | Error comment indicating that `MaxInMemLogSize` may cause `ErrSystemBusy` | [request.go#L64-L66](https://github.com/lni/dragonboat/blob/master/request.go#L64-L66) |

## Summary

- `MaxInMemLogSize` acts as a **soft limit** on unapplied Raft log entries, rejecting new proposals via `ErrPayloadTooBig` when the buffer would exceed the configured size.
- The limit is enforced in [`node.go`](https://github.com/lni/dragonboat/blob/main/node.go) by the `payloadTooBig` helper, which adds `settings.EntryNonCmdFieldsSize` (≈48 bytes) to the payload size before comparison.
- **Production deployments** should set the value to at least 130% of the maximum expected payload plus overhead, while ensuring the total fits within 10–20% of available system memory.
- Setting `MaxInMemLogSize` to `0` disables protection entirely, which is only safe for testing.

## Frequently Asked Questions

### What happens when MaxInMemLogSize is exceeded?

When the in-memory log buffer approaches the limit, the `payloadTooBig` function in [`node.go`](https://github.com/lni/dragonboat/blob/main/node.go) returns `true`, causing the node to reject the proposal. The client receives `dragonboat.ErrPayloadTooBig` if the specific command would exceed the remaining budget, or `dragonboat.ErrSystemBusy` if the node is already at capacity. The application should either retry after a delay or increase the limit if memory permits.

### Can I disable MaxInMemLogSize?

Setting `MaxInMemLogSize` to `0` effectively disables the limit by treating it as `math.MaxUint64`. This allows the in-memory log to grow unbounded until the process runs out of memory. This setting should be reserved for testing or development environments only; production workloads must specify a concrete limit to prevent OOM crashes.

### How does MaxInMemLogSize relate to actual memory usage?

`MaxInMemLogSize` limits only the raw byte size of unapplied log entries stored in the Raft buffer. It does not account for Go runtime overhead, state machine memory, network buffers, or other node host structures. Therefore, the total process memory will be significantly higher than the configured limit. As a rule of thumb, ensure the host has at least 5–10× the `MaxInMemLogSize` available for the entire process.

### What is EntryNonCmdFieldsSize and why does it matter?

`settings.EntryNonCmdFieldsSize` is a constant (approximately 48 bytes) representing the metadata overhead of a Raft log entry, including fields like the term and index. When `payloadTooBig` checks a proposal in [`node.go`](https://github.com/lni/dragonboat/blob/main/node.go), it adds this overhead to the user payload size before comparing against `MaxInMemLogSize`. Failing to account for this overhead in your calculations can cause unexpected rejections of valid commands that appear to be under the limit.