How Telegraf's Output Buffer System Works: Architecture and Backpressure Handling

Telegraf isolates metric collection from delivery using a dual-tier output buffer that automatically promotes metrics from memory to disk when thresholds are reached, and signals input plugins to pause when backpressure occurs, preventing data loss while protecting downstream systems.

The Telegraf output buffer system acts as a critical circuit breaker between data collection and remote storage. In the influxdata/telegraf repository, every output plugin receives metrics through the MetricBuffer interface defined in [models/buffer.go](https://github.com/influxdata/telegraf/blob/master/models/buffer.go), which implements a strategy pattern to balance low-latency performance with durable persistence.

Telegraf Output Buffer Architecture

Telegraf utilizes three distinct buffering strategies that work together to manage metric flow. The core implementation follows a hybrid strategy that dynamically switches between storage tiers based on pressure and configuration.

In-Memory Buffer — Defined in [models/buffer_mem.go](https://github.com/influxdata/telegraf/blob/master/models/buffer_mem.go), this is the default hot path for low-latency ingestion. It maintains metrics in a slice-backed circular buffer until flush thresholds are met or memory limits are exceeded.

On-Disk Buffer — Implemented in [models/buffer_disk.go](https://github.com/influxdata/telegraf/blob/master/models/buffer_disk.go), this provides durable FIFO storage when memory pressure exceeds max_buffer_size. It persists metrics to the filesystem, ensuring resilience against process restarts and downstream outages.

Hybrid Strategy — The overarching behavior is specified in [docs/specs/tsd-005-output-buffer-strategy.md](https://github.com/influxdata/telegraf/blob/master/docs/specs/tsd-005-output-buffer-strategy.md), which defines how the core monitors fill levels and promotes overflow from memory to disk transparently.

How Metrics Flow Through the Buffer System

The lifecycle of a metric through the Telegraf output buffer follows a strict pipeline designed to maximize throughput while respecting resource constraints.

1. Metric Creation — Input plugins generate Metric objects and hand them to the Telegraf agent core.
2. Enqueue — The core invokes buffer.Add(metric). If the in-memory buffer has capacity, the metric stores immediately; otherwise, the buffer promotes it to the on-disk tier.
3. Flush Cycle — Output plugins trigger flushes based on flush_interval (default 10s) or flush_size thresholds. They call buffer.Flush(), which drains the memory buffer first, then the disk buffer in strict FIFO order.
4. Backpressure Detection — When an output plugin's Write method returns an error indicating the downstream system cannot accept data, the core marks the buffer as blocked. This blocked state propagates upstream to pause metric collection temporarily.

Backpressure Handling Mechanisms

When downstream systems become slow or unavailable, Telegraf employs multiple backpressure strategies to prevent unbounded memory growth and protect the host system.

Automatic Tier Promotion — When the in-memory buffer reaches max_buffer_size (default 100MiB), the system automatically persists new metrics to the disk buffer configured via max_disk_buffer_size. If both tiers fill completely, the oldest metrics are dropped according to the max_buffer_keep policy, ensuring the agent never exhausts host resources or crashes.

Input Throttling — Upon detecting a blocked buffer, the core invokes Input.Pause() on any input feeding that output. This halts collection until the buffer drains sufficiently and the core calls Input.Resume(). This feedback loop provides soft backpressure without terminating connections or dropping active metrics prematurely.

Graceful Shutdown — During agent termination, Telegraf flushes remaining data from both memory and disk buffers. This guarantees metric persistence unless the disk buffer limit was already exceeded before shutdown, making recovery predictable and safe.

Configuring the Output Buffer

You tune the Telegraf output buffer system through TOML configuration parameters that control flush behavior and storage limits.

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "metrics"

  # Buffer capacity limits

  max_buffer_size = "100MiB"
  max_disk_buffer_size = "500MiB"
  
  # Flush behavior

  flush_interval = "5s"
  flush_size = 5000

The flush_interval controls how frequently the buffer attempts to write batches regardless of size, while flush_size triggers an immediate flush when the batch reaches the specified metric count. Setting max_disk_buffer_size to zero disables disk buffering, forcing pure in-memory operation with stricter backpressure.

Implementing Custom Buffer-Aware Outputs

When writing custom output plugins, you must implement the MetricBuffer interface to participate in the backpressure system. The agent injects the buffer via SetBuffer(), and your Write method must return errors to signal blocking conditions.

// Minimal output plugin respecting the buffer interface
type MyOutput struct {
    buffer telegraf.MetricBuffer
}

// SetBuffer receives the buffer from the Telegraf core
func (o *MyOutput) SetBuffer(buf telegraf.MetricBuffer) {
    o.buffer = buf
}

// Write processes metrics during flush cycles
func (o *MyOutput) Write(metrics []telegraf.Metric) error {
    // Simulate downstream latency or capacity issues
    if err := sendToRemote(metrics); err != nil {
        // Returning an error signals the core to block the buffer
        return err
    }
    return nil
}

The buffer interface requires methods like Add(), Flush(), and SetMetrics(), which the Telegraf core orchestrates to maintain the metric pipeline.

Summary

• Dual-tier storage: Telegraf uses in-memory buffers for speed and on-disk buffers for durability, automatically promoting metrics between tiers based on max_buffer_size thresholds.
• Backpressure propagation: Blocked buffers trigger Input.Pause() to stop collection upstream, preventing memory exhaustion without dropping data prematurely.
• Configurable safety: Parameters like flush_interval, flush_size, and max_disk_buffer_size let operators tune the trade-off between latency, durability, and resource consumption.
• Graceful degradation: When both buffers reach capacity, the system drops oldest metrics first, ensuring the agent remains stable during extended downstream outages.

Frequently Asked Questions

What happens when the Telegraf output buffer fills up completely?

When both the in-memory and disk buffers reach their configured limits, the buffer implementation begins dropping the oldest metrics to make room for new data. This behavior prevents the Telegraf process from consuming unlimited system memory or disk space. According to the specification in docs/specs/tsd-005-output-buffer-strategy.md, the drop policy is configurable via max_buffer_keep, allowing you to prioritize recent data over historical data during high-pressure scenarios.

How do I configure the Telegraf output buffer for high-throughput environments?

For high-throughput deployments, increase max_buffer_size to accommodate burst writes and reduce flush_interval to ensure frequent delivery attempts. Set flush_size to match your downstream system's optimal batch size, typically between 1,000 and 10,000 metrics. Enable max_disk_buffer_size only if you need crash resilience, as disk I/O can introduce latency; for pure speed, set it to zero to force memory-only buffering with stricter backpressure.

Does Telegraf output buffer persist data across restarts?

Data stored in the on-disk buffer persists across process restarts and system reboots, allowing Telegraf to recover and flush queued metrics upon startup. However, metrics held only in the in-memory buffer are lost if the process terminates unexpectedly. The disk buffer implementation in models/buffer_disk.go handles recovery by reading any existing queue files during plugin initialization before accepting new metrics.

What is the difference between flush_interval and flush_size in Telegraf buffering?

flush_interval defines the maximum time Telegraf waits between flush attempts, ensuring data flows regularly even during low-volume periods. flush_size defines the minimum number of metrics required to trigger an immediate flush, optimizing throughput by sending larger batches. When either condition is satisfied, the buffer drains to the output plugin, making these parameters complementary controls for balancing latency against write efficiency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}