How to Write Custom Parsers for Specific Data Formats in Telegraf

Telegraf custom parsers are Go plugins that implement the telegraf.Parser interface—defining Parse, ParseLine, and SetDefaultTags methods—and register themselves via the parsers.Add function in the global registry.

Telegraf, the open-source server agent from InfluxData, collects and reports metrics from diverse sources. While it ships with built-in parsers for JSON, CSV, and InfluxDB line protocol, many environments require ingesting proprietary or legacy data formats. This guide demonstrates how to write custom parsers for specific data formats in Telegraf using the official plugin architecture found in the influxdata/telegraf repository.

The Parser Interface Contract

Every custom parser must satisfy the telegraf.Parser interface defined in parser.go (lines 5-25). This contract ensures Telegraf can interchangeably use any parser without special-casing logic:

• Parse(buf []byte) ([]telegraf.Metric, error) – Receives a byte slice containing multiple lines or records, decodes them, and returns a slice of metrics.
• ParseLine(line string) (telegraf.Metric, error) – Handles single-line parsing for line-oriented inputs like log files.
• SetDefaultTags(tags map[string]string) – Stores global tags that Telegraf attaches to every metric emitted by this parser.

Step-by-Step Implementation Guide

1. Create the Package Structure

Create a new directory under plugins/parsers/ using your format name as the package identifier:

plugins/parsers/<yourformat>/

Place your implementation in <yourformat>.go within this directory. Use the built-in JSON parser at plugins/parsers/json/parser.go as a reference template for project structure.

2. Define the Parser Struct

Define a struct that holds configuration fields and state. Use TOML tags for configuration fields and always include a telegraf.Logger:

type Parser struct {
    Log     telegraf.Logger `toml:"-"`
    // Add format-specific fields (delimiters, schema paths, etc.)
}

3. Implement the Interface Methods

Implement the three required methods from parser.go. The Parse method typically splits input into lines and delegates to ParseLine:

// Parse processes bulk input and returns a slice of metrics.
func (p *Parser) Parse(buf []byte) ([]telegraf.Metric, error) {
    // Split buffer, iterate lines, call ParseLine, collect results.
}

// ParseLine converts a single string line into a telegraf.Metric.
func (p *Parser) ParseLine(line string) (telegraf.Metric, error) {
    // Decode line and construct metric.
}

// SetDefaultTags stores global tags to attach to every metric.
func (p *Parser) SetDefaultTags(tags map[string]string) {
    p.defaultTags = tags
}

4. Register Your Parser

Add an init function that inserts your parser into the global registry located in plugins/parsers/registry.go (lines 10-16). This enables discovery by name in configuration files:

func init() {
    parsers.Add("<yourformat>", func(defaultMetricName string) telegraf.Parser {
        return &Parser{}
    })
}

Once registered, users can reference your parser in any input plugin:

[[inputs.exec]]
  commands = ["my-binary"]
  data_format = "<yourformat>"

5. Handle Configuration Options

Expose user-configurable options as struct fields with TOML tags. Telegraf automatically unmarshals these from the agent configuration during initialization:

type Parser struct {
    Delimiter string `toml:"delimiter"`
    Log       telegraf.Logger `toml:"-"`
}

Add an Init() method to set default values when fields are omitted:

func (p *Parser) Init() error {
    if p.Delimiter == "" {
        p.Delimiter = ","
    }
    return nil
}

6. Write Tests

Create parser_test.go in your package directory. Use the testutil.ParseMetricsFromFile helper (found in testutil/plugin_input/plugin.go) to verify your parser produces valid telegraf.Metric objects. Follow the testing patterns from plugins/parsers/json/parser_test.go to ensure your implementation handles edge cases correctly.

Complete Custom Parser Example

Below is a minimal, working implementation for a hypothetical csv2 format that parses two-column CSV data. This example demonstrates proper interface implementation and registration:

package csv2

import (
	"bytes"
	"fmt"
	"strings"

	"github.com/influxdata/telegraf"
	"github.com/influxdata/telegraf/plugins/parsers"
)

type Parser struct {
	Log         telegraf.Logger `toml:"-"`
	Delimiter   string          `toml:"delimiter"`
	defaultTags map[string]string
}

func (p *Parser) Init() error {
	if p.Delimiter == "" {
		p.Delimiter = ","
	}
	return nil
}

func (p *Parser) Parse(buf []byte) ([]telegraf.Metric, error) {
	lines := bytes.Split(buf, []byte{'\n'})
	var metrics []telegraf.Metric
	for _, l := range lines {
		if len(l) == 0 {
			continue
		}
		m, err := p.ParseLine(string(l))
		if err != nil {
			return nil, err
		}
		metrics = append(metrics, m)
	}
	return metrics, nil
}

func (p *Parser) ParseLine(line string) (telegraf.Metric, error) {
	parts := strings.SplitN(line, p.Delimiter, 2)
	if len(parts) != 2 {
		return nil, fmt.Errorf("invalid csv line %q", line)
	}
	
	metric := telegraf.NewMetric("csv2_measurement", nil, map[string]string{
		"value": strings.TrimSpace(parts[1]),
	})
	metric.AddTag("field", strings.TrimSpace(parts[0]))
	
	for k, v := range p.defaultTags {
		metric.AddTag(k, v)
	}
	return metric, nil
}

func (p *Parser) SetDefaultTags(tags map[string]string) {
	p.defaultTags = tags
}

func init() {
	parsers.Add("csv2", func(defaultMetricName string) telegraf.Parser {
		return &Parser{}
	})
}

Configuration usage:

[[inputs.exec]]
  commands = ["cat /var/log/myapp.log"]
  data_format = "csv2"
  delimiter = "|"

Key Source Files Reference

• parser.go (lines 5-25) – Defines the telegraf.Parser interface contract.
• plugins/parsers/registry.go (lines 10-16) – Contains the parsers.Add registration function.
• plugins/parsers/json/parser.go – Full-featured reference implementation showing error handling and configuration.
• testutil/plugin_input/plugin.go – Testing utilities for validating parser output.

Summary

• Telegraf parsers are interface-based plugins implementing Parse, ParseLine, and SetDefaultTags.
• Registration via parsers.Add in an init() function makes parsers discoverable by the data_format configuration key.
• Source files belong in plugins/parsers/<format>/ following the established directory convention.
• Configuration fields use TOML tags and are automatically unmarshaled by Telegraf's config loader.
• Testing follows built-in parser patterns using testutil helpers to verify metric generation.

Frequently Asked Questions

What is the difference between the Parse and ParseLine methods?

The Parse method receives a byte slice containing potentially multiple records or lines and returns a slice of telegraf.Metric objects. The ParseLine method handles a single string line and returns exactly one metric. Telegraf calls ParseLine for line-oriented inputs like tailing log files, while Parse is used for bulk data received from network sockets or command output.

How does Telegraf discover my custom parser?

Telegraf discovers parsers through the global registry in plugins/parsers/registry.go. When your package's init() function calls parsers.Add("yourname", constructor), it inserts your parser into a map keyed by name. During configuration loading, Telegraf looks up the string assigned to data_format and instantiates the corresponding parser using the registered constructor function.

Where should I place my custom parser code?

Place your custom parser in a new directory under plugins/parsers/<yourformat>/ within the Telegraf source tree. The package name should match the directory name. This location ensures the Go compiler includes your code when building the Telegraf binary with go build ./..., and it follows the organizational pattern used by the 20+ built-in parsers in the repository.

How do I handle default tags in my parser implementation?

Store the tags map passed to SetDefaultTags in a struct field (typically named defaultTags), then merge these tags into every metric your parser generates. Telegraf calls SetDefaultTags during plugin initialization to provide global tags configured at the agent level, ensuring consistent metadata across all metrics from that parser instance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}