How to Add a New Data Source (Toolset) to HolmesGPT

You add a new data source to HolmesGPT by creating a YAML toolset definition that describes the API or command, registering it via the global config or --custom-toolsets CLI flag, and running holmes toolset refresh to load it into the tool cache.

HolmesGPT discovers problems by executing toolsets—collections of thin wrappers around HTTP APIs, CLI commands, or shell scripts. Extending HolmesGPT to query a new internal API or custom database requires no code changes; you only need to author a declarative YAML file and point HolmesGPT to it. The core implementation lives in holmes/core/toolset_manager.py, with configuration handling in holmes/config.py and CLI parsing in holmes/main.py.

Create a YAML Toolset Definition

A toolset is a YAML map under the top-level key toolsets. The schema is enforced by load_toolsets_from_config in holmes/core/toolset_manager.py and documented in docs/data-sources/custom-toolsets.md.

Place the file anywhere on the host—for example, ~/holmes-toolsets/my-source.yaml:

toolsets:
  my-source:
    description: "Query My Source API for inventory data"
    prerequisites: "API endpoint reachable from Holmes, API key set in env"
    tags: [inventory, api]
    installation: |
      1. Export MY_SOURCE_TOKEN with a read-only API token
      2. Verify network connectivity to https://api.my-source.example.com
    tools:
      - name: list_items
        description: "List all items in the inventory"
        command: |
          curl -s "https://api.my-source.example.com/v1/items" \
               -H "Authorization: Bearer ${MY_SOURCE_TOKEN}"
      - name: get_item
        description: "Fetch details for a single item"
        command: |
          curl -s "https://api.my-source.example.com/v1/items/{{ item_id }}" \
               -H "Authorization: Bearer ${MY_SOURCE_TOKEN}"

Variable syntax:

• {{ variable }} – Inferred by the LLM from the user prompt (e.g., item_id).
• ${VARIABLE} – Read from the container environment; not exposed to the LLM.

Register the Toolset

HolmesGPT supports two registration modes: stable (persistent) and experimental (ad-hoc).

Stable Registration via Config

Add the absolute path to custom_toolsets in your global configuration file (~/.holmes/config.yaml or a project-specific config.yaml):

custom_toolsets:
  - /home/you/holmes-toolsets/my-source.yaml

The Config.custom_toolsets field is defined in holmes/config.py (line 95).

Experimental Registration via CLI

For one-off usage without editing config files, pass the path via the --custom-toolsets flag defined in holmes/main.py (line 103):

holmes ask "list all items from My Source" \
      --custom-toolsets /home/you/holmes-toolsets/my-source.yaml

This populates Config.custom_toolsets_from_cli and takes precedence over stable configurations.

Refresh the Toolset Cache

After registration, refresh the cache so ToolsetManager.load_custom_toolsets() (line 568 in holmes/core/toolset_manager.py) validates and loads the YAML:

holmes toolset refresh

This command invokes _load_toolsets_from_paths, which merges built-in toolsets with your custom definitions. Validation errors—such as missing required fields or invalid YAML—are caught here and reported in the terminal output. Unit tests in tests/core/test_toolset_manager.py cover these validation paths.

Use the New Tools

Once cached, invoke tools by describing them in natural language:

holmes ask "show me the details of item 42 from My Source" \
      --custom-toolsets /home/you/holmes-toolsets/my-source.yaml

The LLM infers item_id=42, selects the get_item tool, substitutes the variable, and returns the API response. No additional code changes are required.

Advanced Topics

Adding Binaries or Scripts

If your toolset requires a binary not present in the base HolmesGPT image (e.g., a proprietary CLI), extend the Docker image as described in docs/data-sources/custom-toolsets.md (lines 45–65). Reference the custom image in your Helm values:

holmes:
  image:
    repository: your-registry/holmesgpt-custom
    tag: latest

Tag-Based Discovery

The optional tags list enables HolmesGPT to filter toolsets when a user query implies a specific domain (e.g., "inventory" or "kubernetes"). Tags are consumed by the runbook selector in holmes/core/investigation.py to narrow tool selection.

Validation and Testing

• YAML validation occurs in load_toolsets_from_config within holmes/core/toolset_manager.py.
• Unit tests in tests/core/test_toolset_manager.py verify loading, error handling, and schema enforcement.
• When developing locally, run make test-without-llm to ensure your configuration parses correctly without incurring LLM API costs.

Summary

• Define a YAML toolset with toolsets, description, tools, and optional prerequisites/installation fields.
• Register the file path either in config.yaml under custom_toolsets for persistent use, or via --custom-toolsets for ad-hoc queries.
• Refresh the cache with holmes toolset refresh to validate and load the new definitions.
• Invoke tools naturally in holmes ask queries; the LLM handles variable substitution and tool selection.

Frequently Asked Questions

Where does HolmesGPT store custom toolset definitions?

HolmesGPT does not store the YAML content internally; it reads the file path you provide at runtime. The ToolsetManager in holmes/core/toolset_manager.py loads and parses the YAML from the absolute paths specified in config.yaml or the --custom-toolsets CLI flag, then caches the validated Toolset objects in memory for the duration of the session.

Can I use environment variables inside tool commands?

Yes. Use ${VARIABLE_NAME} syntax for values that should be injected from the HolmesGPT container’s environment (such as API tokens or database passwords). These variables are resolved by the shell before execution and are never exposed to the LLM. Use {{ variable_name }} syntax for parameters that the LLM should infer from the user’s natural language prompt.

What happens if my YAML toolset definition has a syntax error?

The load_toolsets_from_config function in holmes/core/toolset_manager.py validates the YAML schema when you run holmes toolset refresh or execute a query. If a required field is missing or the YAML is malformed, HolmesGPT prints a descriptive error message to the terminal and skips loading that specific toolset, preventing the invalid configuration from breaking the entire application. You can verify your syntax locally by running make test-without-llm.

Do I need to restart HolmesGPT after adding a new toolset?

No. HolmesGPT loads toolsets dynamically. After adding or modifying a YAML file, run holmes toolset refresh to update the cache. The ToolsetManager will reload the definitions from disk without requiring a process restart. If you use the --custom-toolsets CLI flag, the refresh happens automatically on the first query that references the new tools.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}