How to Create Custom Alerting Rules with Threshold Conditions and Notification Channels in INFINI Console

To create custom alerting rules with threshold conditions in INFINI Console, define a Rule struct with Resource, Metric, and Condition objects, then attach a NotificationConfig containing email or webhook channels before saving via the API or orm.Save().

The INFINI Console alerting subsystem allows you to monitor Elasticsearch clusters and other resources by creating custom alerting rules with threshold conditions. When you create custom alerting rules with threshold conditions, the engine evaluates metric expressions against your defined boundaries and triggers notifications through configurable email or webhook channels.

Understanding the Alerting Rule Architecture

The alerting feature centers on the alerting.Rule model defined in model/alerting/rule.go. A complete rule combines four independent components that work together to evaluate data and dispatch alerts.

Core Components

• Resource (alerting.Resource): Specifies the data source, typically an Elasticsearch cluster ID and type.
• Metric Definition (alerting.Metric): Defines how raw data is fetched, aggregated, and bucketed using fields like cpu.percent with aggregations such as avg.
• Conditions (alerting.Condition and alerting.ConditionItem): Contains threshold logic using operators like gt, lt, gte, lte, equals, or range.
• Notification Configuration (alerting.NotificationConfig): Maps alert severities to specific channels, supporting both normal and escalation workflows.

When Enabled is set to true, the system registers a scheduled task via task.ScheduleTask that periodically executes the rule's query and evaluates conditions.

Defining Threshold Conditions

Threshold conditions determine when your alerting rule fires. The alerting.Condition struct contains a list of ConditionItem objects that support multiple comparison operators.

Condition Operators and Expression Generation

Each ConditionItem in model/alerting/condition.go supports these operators:

• gt (greater than)
• lt (less than)
• gte (greater than or equal)
• lte (less than or equal)
• equals (exact match)
• range (between values)

The GenerateConditionExpression() method converts these items into executable expressions:

func (cond *ConditionItem) GenerateConditionExpression() (conditionExpression string, err error) {
    switch cond.Operator {
    case "gt":
        conditionExpression = fmt.Sprintf("result > %v", cond.Values[0])
    case "lt":
        conditionExpression = fmt.Sprintf("result < %v", cond.Values[0])
    case "range":
        conditionExpression = fmt.Sprintf("result >= %v && result <= %v", cond.Values[0], cond.Values[1])
    // ... additional operators
    }
    return
}

When saving a rule, the system calls Metrics.GenerateExpression() first, then builds each condition expression. The placeholder result is replaced with the actual metric expression via Rule.GetOrInitExpression, storing the final evaluable expression in Rule.Expression.

Configuring Email and Webhook Notification Channels

Notification channels define where alerts are sent when threshold conditions are met. The system supports both immediate notifications and escalation workflows.

Channel Structure

Channel definitions reside in model/alerting/destination.go (lines 34-44):

type Channel struct {
    orm.ORMObjectBase
    Name     string         `json:"name"`
    Type     string         `json:"type"`    // "email" or "webhook"
    Priority int            `json:"priority,omitempty"`
    Webhook  *CustomWebhook `json:"webhook,omitempty"`
    Email    *Email         `json:"email,omitempty"`
    Enabled  bool           `json:"enabled"`
}

• Email channels contain recipient addresses, subject templates, and body templates
• Webhook channels specify URLs, HTTP methods, headers, and authentication via CustomWebhook

Notification Configuration

The NotificationConfig struct in model/alerting/rule.go (lines 100-110) organizes channels into two lists:

• Normal: Channels used for initial alert notifications
• Escalation: Channels used when alerts escalate (e.g., after repeated threshold breaches)

Both lists are optional. If omitted, the rule uses the default notification configuration. The Enabled flag controls whether notifications are sent at all.

Creating a Complete Alerting Rule

You can create custom alerting rules via the REST API or programmatically using Go structs.

JSON Payload for API

Send a POST request to /api/alerting/rules with the following structure:

{
  "name": "High CPU usage",
  "enabled": true,
  "resource": {
    "type": "elasticsearch",
    "id": "es-cluster-01"
  },
  "metrics": {
    "items": [
      {
        "field": "cpu.percent",
        "aggregation": "avg"
      }
    ],
    "group_by": [],
    "bucket_size": "1m"
  },
  "conditions": {
    "operator": "or",
    "items": [
      {
        "operator": "gt",
        "values": ["80"],
        "priority": "high"
      }
    ]
  },
  "notification_config": {
    "enabled": true,
    "title": "CPU Alert – {{rule_name}}",
    "message": "CPU usage is {{value}}% on {{cluster_name}}",
    "normal": [
      {
        "type": "email",
        "email": {
          "to": ["ops@example.com"]
        },
        "enabled": true
      },
      {
        "type": "webhook",
        "webhook": {
          "url": "https://hooks.example.com/alert"
        },
        "enabled": true
      }
    ],
    "escalation": [
      {
        "type": "email",
        "email": {
          "to": ["pagerduty@example.com"]
        },
        "enabled": true,
        "priority": 1
      }
    ]
  },
  "schedule": {
    "interval": "1m"
  }
}

Go Implementation

Create the same rule programmatically:

package main

import (
    "time"
    "github.com/infini.sh/framework/core/orm"
    "infini.sh/console/model/alerting"
)

func main() {
    rule := alerting.Rule{
        Name:    "High CPU usage",
        Enabled: true,
        Resource: alerting.Resource{
            Type: "elasticsearch",
            ID:   "es-cluster-01",
        },
        Metrics: alerting.Metric{
            Items: []alerting.MetricItem{
                {Field: "cpu.percent", Aggregation: "avg"},
            },
            BucketSize: "1m",
        },
        Conditions: alerting.Condition{
            Operator: "or",
            Items: []alerting.ConditionItem{
                {
                    Operator: "gt",
                    Values:   []string{"80"},
                    Priority: "high",
                },
            },
        },
        NotificationConfig: &alerting.NotificationConfig{
            Enabled: true,
            Title:   "CPU Alert – {{rule_name}}",
            Message: "CPU usage is {{value}}% on {{cluster_name}}",
            Normal: []alerting.Channel{
                {
                    Type: alerting.ChannelEmail,
                    Email: &alerting.Email{
                        To: []string{"ops@example.com"},
                    },
                    Enabled: true,
                },
                {
                    Type: alerting.ChannelWebhook,
                    Webhook: &alerting.CustomWebhook{
                        URL: "https://hooks.example.com/alert",
                    },
                    Enabled: true,
                },
            },
            Escalation: []alerting.Channel{
                {
                    Type: alerting.ChannelEmail,
                    Email: &alerting.Email{
                        To: []string{"pagerduty@example.com"},
                    },
                    Enabled: true,
                    Priority: 1,
                },
            },
        },
        Schedule: alerting.Schedule{
            Interval: "1m",
        },
        Created: time.Now(),
        Updated: time.Now(),
    }

    // Generate metric expression
    var err error
    rule.Metrics.Expression, err = rule.Metrics.GenerateExpression()
    if err != nil {
        panic(err)
    }

    // Persist rule
    if err = orm.Save(nil, &rule); err != nil {
        panic(err)
    }
}

Testing and Validation

Before deploying rules to production, validate them using the built-in test functionality.

API Testing Endpoint

Send a POST request to /api/alerting/rules/{id}/test to trigger a single execution:

curl -X POST "http://localhost:8080/api/alerting/rules/<rule-id>/test?type=notification" \
     -H "Content-Type: application/json" \
     -d '{"id":"<rule-id>"}'

The sendTestMessage handler in plugin/api/alerting/rule.go (lines 91-106) invokes eng.Test(&rule, typ), which executes the metric query, evaluates threshold conditions, and dispatches test notifications to configured channels. The response includes ActionExecutionResult items showing which channels succeeded or failed.

Summary

• Alerting rules in INFINI Console combine resources, metrics, conditions, and notification configurations defined in model/alerting/rule.go.
• Threshold conditions use operators like gt, lt, and range to generate executable expressions via GenerateConditionExpression() in model/alerting/condition.go.
• Notification channels support both email and webhook destinations, configured through model/alerting/destination.go and attached to rules via NotificationConfig.
• The API endpoint POST /api/alerting/rules creates rules, while POST /api/alerting/rules/{id}/test validates them before production deployment.
• When Enabled is true, the engine registers a scheduled task that periodically evaluates Rule.Expression and dispatches alerts through the configured channels.

Frequently Asked Questions

How do I configure multiple threshold conditions in a single alerting rule?

You can define multiple ConditionItem objects within the Conditions struct and set the Operator field to "and" or "or" to control logical grouping. Each item supports comparison operators like gt, lt, gte, lte, equals, and range. The engine combines these using GenerateConditionExpression() to build a composite expression that evaluates against the metric results.

What is the difference between normal and escalation notification channels?

Normal channels in NotificationConfig.Normal receive alerts immediately when threshold conditions are first breached. Escalation channels in NotificationConfig.Escalation trigger after repeated violations or when an alert remains active for a specified duration, allowing you to route urgent issues to PagerDuty or management channels while keeping routine notifications in standard email or webhook endpoints.

How do I test if my webhook notification channel is configured correctly?

Use the test API endpoint POST /api/alerting/rules/{id}/test?type=notification with your rule ID. The sendTestMessage handler in plugin/api/alerting/rule.go invokes the engine's Test() method, which executes a single evaluation cycle and attempts to dispatch notifications to all configured channels. The response includes ActionExecutionResult objects indicating success or failure for each channel, allowing you to verify webhook URLs, authentication headers, and email addresses before enabling the rule in production.

Can I use custom expressions instead of simple threshold comparisons?

Yes, while the standard approach uses ConditionItem objects with operators like gt and lt, the system stores the final evaluable expression in Rule.Expression. Advanced users can manipulate this field directly after calling Metrics.GenerateExpression() and GenerateConditionExpression(), combining metric results with custom logic. The engine evaluates this expression using the Engine interface in service/alerting/engine.go, allowing complex conditions like (cpu_avg > 80 AND memory_avg > 90) OR disk_usage > 95.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}