Webhook Notification System Architecture in the INFINI Console Alerting Module

The webhook notification system architecture in INFINI Console implements a pipeline-based design that transforms triggered alerts into HTTP POST requests through discrete components including channel resolution, template rendering, and asynchronous execution.

The alerting subsystem within the infinilabs/console repository provides a robust webhook notification system architecture that enables external integrations. This system processes alert events through a structured pipeline, converting rule violations into formatted HTTP requests dispatched to configurable endpoints.

Overview of the Webhook Notification Pipeline

The architecture follows a five-stage pipeline pattern that separates concerns between configuration storage, runtime resolution, action execution, HTTP dispatch, and result collection.

1. Configuration Storage – Webhook destinations persist as Channel entities with Type = "webhook".
2. Runtime Resolution – The engine retrieves channels and resolves templated values in URLs and bodies.
3. Action Execution – Resolved data instantiates a concrete WebhookAction.
4. HTTP Dispatch – The action constructs and sends HTTP POST requests via a shared http.Client.
5. Result Collection – Response data aggregates into ActionExecutionResult records for audit trails.

Each stage is handled by specialized components within the service/alerting package, ensuring clean separation between data models, business logic, and transport mechanisms.

Core Components and Data Models

Channel Configuration

Webhook destinations are persisted as Channel entities defined in model/alerting/destination.go. Each channel specifies a Type field set to "webhook" and embeds a CustomWebhook configuration struct containing the endpoint details.

Webhook Payload Structure

The CustomWebhook struct in model/alerting/webhook.go defines the HTTP request parameters:

• Method: HTTP verb (typically POST)
• URL: Endpoint address supporting Go text/template syntax
• HeaderParams: Custom HTTP headers for authentication or content-type
• Body: Request payload with template variables

Runtime Execution Flow

Channel Retrieval

When an alert triggers, the engine invokes common.RetrieveChannel from service/alerting/common/helper.go. This function loads the persisted channel configuration, validates that the channel is enabled, and populates default values for missing fields.

Template Resolution

The common.ResolveMessage function processes the webhook URL and Body fields as Go text/template strings. It injects alert-specific context variables including title, message, rule_name, and environment variables from the ctx map.

Action Dispatch

The resolved configuration is passed to common.PerformChannel, which instantiates a WebhookAction from service/alerting/action/webhook.go. The action's Execute method constructs the HTTP request, applies custom headers, and dispatches the POST request using a shared http.Client.

Result Aggregation

The performChannels function in service/alerting/elasticsearch/engine.go orchestrates the entire workflow. It iterates over configured channels, invokes PerformChannel for each, and aggregates the results into ActionExecutionResult structs that are persisted with the alert record.

Implementation Example

The following example demonstrates defining and executing a webhook channel:

// Define a webhook channel configuration
channel := alerting.Channel{
    Name: "Production Slack Alerts",
    Type: alerting.ChannelWebhook,
    Webhook: &alerting.CustomWebhook{
        Method: "POST",
        URL:    "https://hooks.slack.com/services/{{.env.SLACK_WEBHOOK}}",
        HeaderParams: map[string]string{
            "Content-Type": "application/json",
        },
        Body: `{
            "text": "{{.title}} - {{.message}}",
            "attachments": [
                {"title": "{{.rule_name}}", "color": "danger"}
            ]
        }`,
    },
    Enabled: true,
}

// Simulate alert context
ctx := map[string]interface{}{
    "title":     "High CPU Usage",
    "message":   "CPU utilization exceeded 90% on node-1",
    "rule_name": "CPU_Critical",
    "env": map[string]string{
        "SLACK_WEBHOOK": "T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
    },
}

// Execute the webhook
result, err, _ := common.PerformChannel(&channel, ctx)
if err != nil {
    log.Fatalf("Webhook delivery failed: %v", err)
}
log.Printf("Webhook response: %s", string(result))

The same logic is invoked automatically when an alert rule matches, as part of the Engine.Do workflow in service/alerting/elasticsearch/engine.go.

Key Source Files

File	Responsibility
model/alerting/destination.go	Defines the Channel struct and webhook configuration storage
model/alerting/webhook.go	Contains the CustomWebhook payload structure
service/alerting/common/helper.go	Implements RetrieveChannel, ResolveMessage, and PerformChannel
service/alerting/action/webhook.go	Houses the WebhookAction HTTP client implementation
service/alerting/elasticsearch/engine.go	Orchestrates alert processing and channel execution
plugin/api/alerting/channel.go	Provides the REST API endpoint for testing webhook channels

Summary

• The webhook notification system architecture in INFINI Console employs a pipeline pattern separating configuration, resolution, and execution concerns.
• Channel configurations are stored as Channel entities with embedded CustomWebhook structs in model/alerting/destination.go.
• Template resolution occurs via common.ResolveMessage, which processes Go text/template syntax in URLs and request bodies.
• HTTP dispatch is handled by WebhookAction.Execute in service/alerting/action/webhook.go, using a shared http.Client for connection reuse.
• The alert engine in service/alerting/elasticsearch/engine.go orchestrates the entire workflow, aggregating results into ActionExecutionResult records.

Frequently Asked Questions

How does the webhook notification system handle authentication headers?

The system supports custom HTTP headers through the HeaderParams field in the CustomWebhook struct. You can define authentication tokens, API keys, or content-type specifications in this map, and WebhookAction.Execute applies these headers to the outgoing HTTP request before dispatch.

Can webhook URLs and payloads use dynamic variables from the alert context?

Yes. The common.ResolveMessage function in service/alerting/common/helper.go treats both the URL and Body fields as Go text/template strings. It injects alert-specific variables such as title, message, rule_name, and environment variables from the ctx map, enabling dynamic endpoint targeting and personalized payloads.

What happens if a webhook delivery fails?

When WebhookAction.Execute encounters an HTTP error or non-success status code, it returns the error to the caller. The performChannels function in service/alerting/elasticsearch/engine.go captures these results in ActionExecutionResult structs, which are persisted alongside the alert record. This allows operators to review delivery failures through the alert history API.

Is there a way to test webhook configurations before enabling them?

Yes. The AlertAPI.testChannel handler in plugin/api/alerting/channel.go exposes a REST endpoint at /alert/channel/test. This endpoint constructs a sample alert context and invokes common.PerformChannel against the provided webhook configuration, allowing administrators to verify connectivity and payload formatting without triggering actual alert conditions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}