How to Debug Telegraf Plugin Loading and Configuration Issues: A Complete Guide
Enable agent debug mode with debug = true in the [agent] section and run telegraf --config telegraf.conf --debug to trace the registration, parsing, and construction phases while monitoring config/config.go for field-miss tracking and plugin factory lookups.
Debugging Telegraf plugin loading and configuration issues requires understanding how the influxdata/telegraf repository discovers and instantiates plugins from TOML configuration. The initialization process involves three distinct phases—registration, parsing, and construction—each of which can fail silently or emit cryptic errors without proper diagnostic flags. By instrumenting the agent debug mode and inspecting specific files like config/config.go, you can isolate whether failures occur during factory registration, AST parsing, or runtime wiring.
Understanding the Three Phases of Plugin Initialization
Telegraf discovers, loads, and configures plugins (inputs, processors, aggregators, outputs, secret-stores, and parsers) through a rigid three-phase pipeline implemented in the core configuration engine.
Phase 1: Plugin Registration via init()
Each plugin package registers a factory function in a global map during package initialization. These maps—inputs.Inputs, processors.Processors, outputs.Outputs, and others—are populated inside init() blocks when the binary starts.
For example, in plugins/inputs/cpu/cpu.go, the CPU input registers itself with:
inputs.Add("cpu", func() telegraf.Input {
return &CPUStats{}
})
This pattern applies to every built-in plugin and any custom plugins compiled into the Telegraf binary. If the init() function does not execute, the factory remains absent from the global map, causing "undefined plugin" errors later.
Phase 2: TOML Configuration Parsing
When you run telegraf --config telegraf.conf, the entry point calls config.Config.LoadConfig (defined in config/config.go), which chains to LoadConfigData and finally parseConfig. This sequence:
- Builds an Abstract Syntax Tree (AST) using
github.com/influxdata/toml/ast.Table. - Walks each top-level key (
agent,inputs,outputs,processors, etc.). - Looks up the plugin name in the registration map (e.g.,
inputs.Inputs[name]) to retrieve the factory function.
If the TOML contains syntax errors, Telegraf aborts here before any plugin instantiation occurs.
Phase 3: Plugin Construction and Wiring
The helper methods addInput, addOutput, addProcessor, addAggregator, and addSecretStore (all in config/config.go) perform the final assembly:
- Label selection: Evaluates optional
labelfilters (InputFilters,OutputFilters, etc.) viamatchesLabelSelection. - Parser/serializer wiring: If a plugin implements
ParserPluginorSerializerPlugin, the methodsaddParserandaddSerializerattach the appropriate codec. - Field-miss tracking: A local counter records TOML fields not consumed during unmarshalling; excessive misses trigger configuration errors.
- TLS validation: For outputs exposing
TLSConfig(), the method validates certificates and TLS settings. - Secret linking: After all plugins instantiate,
LinkSecretsresolves references like${INFLUX_TOKEN}against secret-stores.
Enabling Verbose Debugging Mode
The most direct way to debug Telegraf plugin loading and configuration issues is to enable agent debug mode.
Add this to your configuration file:
[agent]
debug = true
When debug is true, Telegraf logs every loading step, including:
- "Loading config: …" emitted by
LoadConfig. - "I! Plugin … added" printed by the
add*functions after successful unmarshalling. - Warnings for unknown fields, TLS errors, or deprecation notices.
Alternatively, enable debugging temporarily from the command line without modifying the config:
telegraf --config telegraf.conf --debug
For maximum verbosity including stack traces, set the logger to the trace level:
[agent]
logformat = "structured"
logtarget = "stderr"
debug = true
Diagnosing Common Plugin Loading Failures
When a plugin fails to load, Telegraf emits specific error messages originating from distinct locations in config/config.go:
| Situation | Error Message | Origin |
|---|---|---|
| Unknown plugin name | undefined but requested input: xyz |
addInput, addOutput, etc., when the name is absent from the registration map. |
| Missing required fields | line X: configuration specified the fields "foo", but they were not used |
c.missingTomlField tracking unused TOML keys after unmarshalling. |
| TLS mis-configuration | Errors returned from TLSConfig() validation |
addOutput when TLS certificate checks fail. |
| Deprecated plugin | plugin deprecated notice |
printHistoricPluginDeprecationNotice checking inputs.Deprecations or outputs.Deprecations maps. |
| Secret-store missing | unknown secret-store |
LinkSecrets when a ${secret:store:id} reference cannot be resolved. |
Step-by-Step Debugging Workflow
Follow this systematic approach to isolate plugin loading failures:
-
Verify registration: Confirm the plugin's
init()runs by executinggo test ./... -run TestPluginRegisteredin the plugin's package. If the test fails, the plugin is not compiled into the binary. -
Validate TOML syntax: Use
toml-lintor runtelegraf --config /dev/nullto isolate syntax errors that prevent parsing from reaching the plugin initialization phase. -
Inspect the parsed AST: Temporarily insert
log.Printf("%#v", tbl)after theparseConfigcall inconfig/config.goto view the exact structure Telegraf received from the TOML parser. -
Confirm filter lists: If using
namepass/namedropor the--input-filterflag, ensure the target plugin name matches exactly (case-sensitive) against the registered name. -
Locate unused fields: Check the line number in "field not used" errors. The error message cites the specific line in the original file (e.g.,
line 27) where Telegraf detected unconsumed configuration keys. -
Check TLS certificates: For outputs requiring TLS (e.g.,
outputs.influxdb_v2), verify certificate file paths and setinsecure_skip_verify = truetemporarily to rule out certificate validation issues. -
Validate secret references: Ensure secret-store IDs in
${secret:store:id}syntax match defined stores exactly; missing definitions causeLinkSecretsto fail after plugin construction.
Practical Debugging Examples
Minimal Config with Debug Enabled
Create telegraf.conf:
[agent]
debug = true
interval = "10s"
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = true
[[outputs.influxdb]]
urls = ["http://localhost:8086"]
database = "telegraf"
Run with:
telegraf --config ./telegraf.conf
Expected debug output:
I! Loading config: ./telegraf.conf
I! Adding input cpu
I! Adding output influxdb
D! Input cpu initialized
D! Output influxdb initialized
Verifying Custom Plugin Registration
For a custom plugin compiled into the binary:
// myplugin.go
package myplugin
import (
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/plugins/inputs"
)
type MyPlugin struct{}
func (p *MyPlugin) SampleConfig() string { return "" }
func (p *MyPlugin) Gather(acc telegraf.Accumulator) error { return nil }
func init() {
inputs.Add("myplugin", func() telegraf.Input { return &MyPlugin{} })
}
After building:
go build -o telegraf ./cmd/telegraf
go tool nm telegraf | grep myplugin
If the symbol appears, registration succeeded. If not, the package import is missing from your build.
Detecting Configuration Field Errors
Given this config:
[[inputs.cpu]]
unknown_option = true
Telegraf outputs:
E! line 2: configuration specified the fields "unknown_option", but they were not used; this is either a typo or this config option does not exist in this version
The line number points directly to the offending key in plugins/inputs/cpu/cpu.go or your custom TOML.
Summary
- Plugin registration occurs in
init()blocks usinginputs.Addand similar functions; verify withgo testorgo tool nm. - Configuration parsing happens in
config/config.goviaLoadConfig→parseConfig, building an AST that must match registered plugin names. - Construction and wiring uses
addInput,addOutput, and related methods to handle labels, parsers, TLS, and secret resolution. - Enable
debug = truein the[agent]section or use--debugto trace the exact phase where loading fails. - Field-miss tracking in
config/config.gocatches typos by reporting unused TOML fields with specific line numbers.
Frequently Asked Questions
How do I know if a plugin is registered correctly?
Run go test ./... -run TestPluginRegistered within the plugin's directory to verify the init() function executed. Alternatively, after building Telegraf, use go tool nm telegraf | grep <plugin_name> to confirm the registration symbol exists in the binary; if absent, the plugin package is not imported in your build.
Why does Telegraf say a field is not used?
This error originates from the field-miss counter in config/config.go. After unmarshalling TOML into a plugin struct, Telegraf compares consumed fields against the AST; any field present in the config but absent from the struct (or misspelled) increments the miss count. The error cites the exact line number where the unused field appears.
How can I see the exact config structure Telegraf parses?
Insert a temporary debug line after the parseConfig function in config/config.go to print the AST table: log.Printf("%#v", tbl). This outputs the raw structure derived from your TOML, revealing plugin names, nested tables, and key-value pairs exactly as the parser interpreted them.
What causes "undefined but requested input" errors?
This message emits from addInput, addOutput, or similar methods when the plugin name specified in your TOML does not exist in the global registration map (e.g., inputs.Inputs). Causes include typos in the plugin name, missing imports for custom plugins, or attempting to use a plugin excluded from your specific Telegraf build.
Have a question about this repo?
These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:
curl -s "https://instagit.com/install.md" Maintain an open-source project? Get it listed too →