# How to Create Custom Alerting Rules with Threshold Conditions and Notification Channels in INFINI Console

> Create custom alerting rules in INFINI Console with threshold conditions and configure email or webhook notifications. Master efficient monitoring today.

- Repository: [INFINI Labs/console](https://github.com/infinilabs/console)
- Tags: how-to-guide
- Published: 2026-03-04

---

**To create custom alerting rules with threshold conditions in INFINI Console, define a `Rule` struct with `Resource`, `Metric`, and `Condition` objects, then attach a `NotificationConfig` containing email or webhook channels before saving via the API or `orm.Save()`.**

The INFINI Console alerting subsystem allows you to monitor Elasticsearch clusters and other resources by creating custom alerting rules with threshold conditions. When you create custom alerting rules with threshold conditions, the engine evaluates metric expressions against your defined boundaries and triggers notifications through configurable email or webhook channels.

## Understanding the Alerting Rule Architecture

The alerting feature centers on the **`alerting.Rule`** model defined in [`model/alerting/rule.go`](https://github.com/infinilabs/console/blob/main/model/alerting/rule.go). A complete rule combines four independent components that work together to evaluate data and dispatch alerts.

### Core Components

- **Resource** (`alerting.Resource`): Specifies the data source, typically an Elasticsearch cluster ID and type.
- **Metric Definition** (`alerting.Metric`): Defines how raw data is fetched, aggregated, and bucketed using fields like `cpu.percent` with aggregations such as `avg`.
- **Conditions** (`alerting.Condition` and `alerting.ConditionItem`): Contains threshold logic using operators like `gt`, `lt`, `gte`, `lte`, `equals`, or `range`.
- **Notification Configuration** (`alerting.NotificationConfig`): Maps alert severities to specific channels, supporting both normal and escalation workflows.

When `Enabled` is set to `true`, the system registers a scheduled task via `task.ScheduleTask` that periodically executes the rule's query and evaluates conditions.

## Defining Threshold Conditions

Threshold conditions determine when your alerting rule fires. The `alerting.Condition` struct contains a list of `ConditionItem` objects that support multiple comparison operators.

### Condition Operators and Expression Generation

Each `ConditionItem` in [`model/alerting/condition.go`](https://github.com/infinilabs/console/blob/main/model/alerting/condition.go) supports these operators:

- `gt` (greater than)
- `lt` (less than)
- `gte` (greater than or equal)
- `lte` (less than or equal)
- `equals` (exact match)
- `range` (between values)

The `GenerateConditionExpression()` method converts these items into executable expressions:

```go
func (cond *ConditionItem) GenerateConditionExpression() (conditionExpression string, err error) {
    switch cond.Operator {
    case "gt":
        conditionExpression = fmt.Sprintf("result > %v", cond.Values[0])
    case "lt":
        conditionExpression = fmt.Sprintf("result < %v", cond.Values[0])
    case "range":
        conditionExpression = fmt.Sprintf("result >= %v && result <= %v", cond.Values[0], cond.Values[1])
    // ... additional operators
    }
    return
}

```

When saving a rule, the system calls `Metrics.GenerateExpression()` first, then builds each condition expression. The placeholder `result` is replaced with the actual metric expression via `Rule.GetOrInitExpression`, storing the final evaluable expression in `Rule.Expression`.

## Configuring Email and Webhook Notification Channels

Notification channels define where alerts are sent when threshold conditions are met. The system supports both immediate notifications and escalation workflows.

### Channel Structure

Channel definitions reside in [`model/alerting/destination.go`](https://github.com/infinilabs/console/blob/main/model/alerting/destination.go) (lines 34-44):

```go
type Channel struct {
    orm.ORMObjectBase
    Name     string         `json:"name"`
    Type     string         `json:"type"`    // "email" or "webhook"
    Priority int            `json:"priority,omitempty"`
    Webhook  *CustomWebhook `json:"webhook,omitempty"`
    Email    *Email         `json:"email,omitempty"`
    Enabled  bool           `json:"enabled"`
}

```

- **Email channels** contain recipient addresses, subject templates, and body templates
- **Webhook channels** specify URLs, HTTP methods, headers, and authentication via `CustomWebhook`

### Notification Configuration

The `NotificationConfig` struct in [`model/alerting/rule.go`](https://github.com/infinilabs/console/blob/main/model/alerting/rule.go) (lines 100-110) organizes channels into two lists:

- `Normal`: Channels used for initial alert notifications
- `Escalation`: Channels used when alerts escalate (e.g., after repeated threshold breaches)

Both lists are optional. If omitted, the rule uses the default notification configuration. The `Enabled` flag controls whether notifications are sent at all.

## Creating a Complete Alerting Rule

You can create custom alerting rules via the REST API or programmatically using Go structs.

### JSON Payload for API

Send a POST request to `/api/alerting/rules` with the following structure:

```json
{
  "name": "High CPU usage",
  "enabled": true,
  "resource": {
    "type": "elasticsearch",
    "id": "es-cluster-01"
  },
  "metrics": {
    "items": [
      {
        "field": "cpu.percent",
        "aggregation": "avg"
      }
    ],
    "group_by": [],
    "bucket_size": "1m"
  },
  "conditions": {
    "operator": "or",
    "items": [
      {
        "operator": "gt",
        "values": ["80"],
        "priority": "high"
      }
    ]
  },
  "notification_config": {
    "enabled": true,
    "title": "CPU Alert – {{rule_name}}",
    "message": "CPU usage is {{value}}% on {{cluster_name}}",
    "normal": [
      {
        "type": "email",
        "email": {
          "to": ["ops@example.com"]
        },
        "enabled": true
      },
      {
        "type": "webhook",
        "webhook": {
          "url": "https://hooks.example.com/alert"
        },
        "enabled": true
      }
    ],
    "escalation": [
      {
        "type": "email",
        "email": {
          "to": ["pagerduty@example.com"]
        },
        "enabled": true,
        "priority": 1
      }
    ]
  },
  "schedule": {
    "interval": "1m"
  }
}

```

### Go Implementation

Create the same rule programmatically:

```go
package main

import (
    "time"
    "github.com/infini.sh/framework/core/orm"
    "infini.sh/console/model/alerting"
)

func main() {
    rule := alerting.Rule{
        Name:    "High CPU usage",
        Enabled: true,
        Resource: alerting.Resource{
            Type: "elasticsearch",
            ID:   "es-cluster-01",
        },
        Metrics: alerting.Metric{
            Items: []alerting.MetricItem{
                {Field: "cpu.percent", Aggregation: "avg"},
            },
            BucketSize: "1m",
        },
        Conditions: alerting.Condition{
            Operator: "or",
            Items: []alerting.ConditionItem{
                {
                    Operator: "gt",
                    Values:   []string{"80"},
                    Priority: "high",
                },
            },
        },
        NotificationConfig: &alerting.NotificationConfig{
            Enabled: true,
            Title:   "CPU Alert – {{rule_name}}",
            Message: "CPU usage is {{value}}% on {{cluster_name}}",
            Normal: []alerting.Channel{
                {
                    Type: alerting.ChannelEmail,
                    Email: &alerting.Email{
                        To: []string{"ops@example.com"},
                    },
                    Enabled: true,
                },
                {
                    Type: alerting.ChannelWebhook,
                    Webhook: &alerting.CustomWebhook{
                        URL: "https://hooks.example.com/alert",
                    },
                    Enabled: true,
                },
            },
            Escalation: []alerting.Channel{
                {
                    Type: alerting.ChannelEmail,
                    Email: &alerting.Email{
                        To: []string{"pagerduty@example.com"},
                    },
                    Enabled: true,
                    Priority: 1,
                },
            },
        },
        Schedule: alerting.Schedule{
            Interval: "1m",
        },
        Created: time.Now(),
        Updated: time.Now(),
    }

    // Generate metric expression
    var err error
    rule.Metrics.Expression, err = rule.Metrics.GenerateExpression()
    if err != nil {
        panic(err)
    }

    // Persist rule
    if err = orm.Save(nil, &rule); err != nil {
        panic(err)
    }
}

```

## Testing and Validation

Before deploying rules to production, validate them using the built-in test functionality.

### API Testing Endpoint

Send a POST request to `/api/alerting/rules/{id}/test` to trigger a single execution:

```bash
curl -X POST "http://localhost:8080/api/alerting/rules/<rule-id>/test?type=notification" \
     -H "Content-Type: application/json" \
     -d '{"id":"<rule-id>"}'

```

The `sendTestMessage` handler in [`plugin/api/alerting/rule.go`](https://github.com/infinilabs/console/blob/main/plugin/api/alerting/rule.go) (lines 91-106) invokes `eng.Test(&rule, typ)`, which executes the metric query, evaluates threshold conditions, and dispatches test notifications to configured channels. The response includes `ActionExecutionResult` items showing which channels succeeded or failed.

## Summary

- **Alerting rules** in INFINI Console combine resources, metrics, conditions, and notification configurations defined in [`model/alerting/rule.go`](https://github.com/infinilabs/console/blob/main/model/alerting/rule.go).
- **Threshold conditions** use operators like `gt`, `lt`, and `range` to generate executable expressions via `GenerateConditionExpression()` in [`model/alerting/condition.go`](https://github.com/infinilabs/console/blob/main/model/alerting/condition.go).
- **Notification channels** support both **email** and **webhook** destinations, configured through [`model/alerting/destination.go`](https://github.com/infinilabs/console/blob/main/model/alerting/destination.go) and attached to rules via `NotificationConfig`.
- The **API endpoint** `POST /api/alerting/rules` creates rules, while `POST /api/alerting/rules/{id}/test` validates them before production deployment.
- When `Enabled` is true, the engine registers a scheduled task that periodically evaluates `Rule.Expression` and dispatches alerts through the configured channels.

## Frequently Asked Questions

### How do I configure multiple threshold conditions in a single alerting rule?

You can define multiple `ConditionItem` objects within the `Conditions` struct and set the `Operator` field to `"and"` or `"or"` to control logical grouping. Each item supports comparison operators like `gt`, `lt`, `gte`, `lte`, `equals`, and `range`. The engine combines these using `GenerateConditionExpression()` to build a composite expression that evaluates against the metric results.

### What is the difference between normal and escalation notification channels?

**Normal** channels in `NotificationConfig.Normal` receive alerts immediately when threshold conditions are first breached. **Escalation** channels in `NotificationConfig.Escalation` trigger after repeated violations or when an alert remains active for a specified duration, allowing you to route urgent issues to PagerDuty or management channels while keeping routine notifications in standard email or webhook endpoints.

### How do I test if my webhook notification channel is configured correctly?

Use the test API endpoint `POST /api/alerting/rules/{id}/test?type=notification` with your rule ID. The `sendTestMessage` handler in [`plugin/api/alerting/rule.go`](https://github.com/infinilabs/console/blob/main/plugin/api/alerting/rule.go) invokes the engine's `Test()` method, which executes a single evaluation cycle and attempts to dispatch notifications to all configured channels. The response includes `ActionExecutionResult` objects indicating success or failure for each channel, allowing you to verify webhook URLs, authentication headers, and email addresses before enabling the rule in production.

### Can I use custom expressions instead of simple threshold comparisons?

Yes, while the standard approach uses `ConditionItem` objects with operators like `gt` and `lt`, the system stores the final evaluable expression in `Rule.Expression`. Advanced users can manipulate this field directly after calling `Metrics.GenerateExpression()` and `GenerateConditionExpression()`, combining metric results with custom logic. The engine evaluates this expression using the `Engine` interface in [`service/alerting/engine.go`](https://github.com/infinilabs/console/blob/main/service/alerting/engine.go), allowing complex conditions like `(cpu_avg > 80 AND memory_avg > 90) OR disk_usage > 95`.