# Data Model for Alerting Rules and Condition Evaluation in InfiniLabs Console > Explore the InfiniLabs Console data model for alerting rules. Discover how resource targets, metric expressions, and prioritized conditions evaluate Elasticsearch data using govaluate. - Repository: [INFINI Labs/console](https://github.com/infinilabs/console) - Tags: api-reference - Published: 2026-03-04 --- **The alerting engine in InfiniLabs Console uses a JSON-driven Rule struct that combines resource targets, metric expressions, and prioritized conditions to evaluate Elasticsearch data via govaluate expressions.** The InfiniLabs Console repository (`infinilabs/console`) provides a comprehensive alerting subsystem designed to monitor Elasticsearch clusters. At its core lies a structured data model for alerting rules that supports both numeric thresholds and complex bucket-diff conditions. This article examines the Rule struct definition, condition evaluation logic, and the engine implementation that processes these rules against time-series data. ## Alerting Rule Data Model ### The Rule Struct The foundation of the alerting system is the **`Rule`** struct defined in [`model/alerting/rule.go`](https://github.com/infinilabs/console/blob/main/model/alerting/rule.go). This structure encapsulates all configuration necessary to define, schedule, and execute a monitoring rule: - **Identification and Metadata**: `ID`, `Created`, `Updated`, `Name`, and `Enabled` fields provide basic rule management. - **Resource Targeting**: The `Resource` field (defined in [`model/alerting/resource.go`](https://github.com/infinilabs/console/blob/main/model/alerting/resource.go)) specifies the Elasticsearch indices, time field, and raw filters to query. - **Metric Configuration**: The `Metrics` field (from [`model/alerting/metric.go`](https://github.com/infinilabs/console/blob/main/model/alerting/metric.go)) defines aggregation functions, bucket sizes, and mathematical formulas. - **Condition Logic**: `Conditions` and optional `BucketConditions` (from [`model/alerting/condition.go`](https://github.com/infinilabs/console/blob/main/model/alerting/condition.go)) establish the logical thresholds that trigger alerts. - **Notification Settings**: `Channels`, `NotificationConfig`, and `RecoveryNotificationConfig` control webhook, email, and escalation behaviors. - **Scheduling**: The `Schedule` field (from [`model/alerting/schedule.go`](https://github.com/infinilabs/console/blob/main/model/alerting/schedule.go)) provides cron-like execution timing. - **Runtime State**: Fields like `LastNotificationTime`, `LastEscalationTime`, and `LastTermStartTime` manage throttling and stateful tracking. The struct also provides `Rule.GetOrInitExpression`, which generates a full evaluable expression combining metric formulas with condition expressions. ### Condition Definitions Conditions are defined in [`model/alerting/condition.go`](https://github.com/infinilabs/console/blob/main/model/alerting/condition.go) through two primary structures: **`Condition`** acts as a container with: - `Operator`: Currently supports "or" logic between items - `Items`: A slice of `ConditionItem` structs **`ConditionItem`** represents individual threshold checks with: - `Operator`: Comparison operators (`equals`, `gte`, `lt`, `gt`, `lte`, `neq`) - `Values`: Threshold values as strings - `MinimumPeriodMatch`: Number of consecutive data points that must satisfy the condition - `Priority`: Severity level (affects evaluation order) - `Type` and `BucketCount`: Special fields for bucket-diff conditions The `ConditionItem.GenerateConditionExpression` method produces evaluable expressions using the placeholder variable `result`. For example, a "gte" operator generates: ```go conditionExpression = fmt.Sprintf("result >= %v", cond.Values[0]) ``` ### Metric Expressions The `Metric` struct in [`model/alerting/metric.go`](https://github.com/infinilabs/console/blob/main/model/alerting/metric.go) defines how to extract and compute values from Elasticsearch: - `Items`: List of `MetricItem` structs specifying fields and aggregations (max, avg, percentile, etc.) - `BucketSize`: Time interval for aggregation - `Formula`: Mathematical expression combining metric items (e.g., `"a + b"` or `"result"`) The `Metric.GenerateExpression` method constructs the evaluable formula used by the engine. ## How Conditions Are Evaluated in the Alerting Engine The evaluation logic resides in [`service/alerting/elasticsearch/engine.go`](https://github.com/infinilabs/console/blob/main/service/alerting/elasticsearch/engine.go). The engine processes rules through a pipeline that prepares metric data, evaluates conditions by priority, and manages stateful notifications. ### Data Preparation and Metric Calculation The `engine.GetTargetMetricData` method orchestrates the initial data retrieval: 1. **Query Generation**: Constructs Elasticsearch queries using `GenerateQuery` based on the rule's resource and metric definitions. 2. **Data Retrieval**: Executes the query against the target cluster. 3. **Formula Evaluation**: For rules with multiple metric items, evaluates the user-supplied formula using `govaluate.NewEvaluableExpression`. The result is a slice of `insight.MetricData` where computed values are stored under the key `"result"`. This preparation ensures that subsequent condition evaluation works against normalized `result` values regardless of the underlying metric complexity. ### Bucket Condition Evaluation For rules requiring stateful bucket analysis (such as detecting content changes or document count anomalies), the engine uses `CheckBucketCondition` (lines 600-740 in [`engine.go`](https://github.com/infinilabs/console/blob/main/engine.go)): 1. **Timestamp Collation**: Gathers all unique bucket timestamps across metric groups. 2. **Diff State Calculation**: Computes either `ContentChangeState` (for content-based rules) or `DocCount` (for volume-based rules) for each timestamp. 3. **Expression Evaluation**: Evaluates bucket-condition expressions using `govaluate` against the computed diff values. 4. **Consecutive Period Matching**: Tracks how many consecutive buckets satisfy the condition. When the count reaches `MinimumPeriodMatch`, the condition triggers. This approach supports complex scenarios like "alert if index content changes for 3 consecutive minutes" or "alert if document count drops for 2 consecutive buckets." ### Standard Numeric Condition Checking When no bucket conditions exist, `CheckCondition` (lines 870-990 in [`engine.go`](https://github.com/infinilabs/console/blob/main/engine.go)) handles numeric threshold evaluation: **Priority-Based Ordering**: Conditions are sorted by priority weight to ensure critical alerts evaluate first: ```go sort.Slice(rule.Conditions.Items, func(i, j int) bool { return alerting.PriorityWeights[rule.Conditions.Items[i].Priority] > alerting.PriorityWeights[rule.Conditions.Items[j].Priority] }) ``` **Per-Series Evaluation**: For each metric data series in `targetMetricData`: - Determines the shortest series length and identifies the `dataKey` containing result values. - Iterates through each data point index `i`. - Builds a `relationValues` map containing raw values of all metric items for that timestamp. - Evaluates the metric formula using these values to produce `valueExpressionResult`. - Applies the condition expression (generated from `ConditionItem`) to `valueExpressionResult` using `govaluate`. - Maintains a `triggerCount` for consecutive matches. When `triggerCount` reaches `cond.MinimumPeriodMatch`, the condition is marked triggered and a `ConditionResultItem` is recorded. - Stops checking further conditions for that series once a condition fires (breaks the `LoopCondition`). This design ensures efficient evaluation where high-priority conditions short-circuit lower-priority checks, and multi-period requirements prevent flapping alerts from single-point anomalies. ### Notification and Escalation Handling After condition evaluation completes, the `Do` method (lines 530-730 in [`engine.go`](https://github.com/infinilabs/console/blob/main/engine.go)) manages the alert lifecycle: - **Alert Creation**: Creates or updates `Alert` records based on `ConditionResult`. - **Throttling**: Checks `LastNotificationTime` against notification configuration to prevent spam. - **Escalation**: Tracks `LastEscalationTime` to escalate unacknowledged alerts after configured periods. - **Recovery Detection**: When conditions no longer trigger, sends recovery notifications using `RecoveryNotificationConfig`. - **Channel Dispatch**: Routes messages through configured channels (webhook, email) using template resolution from `service/alerting/common`. ## Practical Implementation Examples ### Defining and Loading a Rule The following JSON structure represents a complete alerting rule configuration targeting high CPU usage: ```json { "name": "CPU usage high", "enabled": true, "resource": { "id": "es-cluster-1", "objects": ["metrics*"], "time_field": "@timestamp" }, "metrics": { "items": [ {"name":"cpu","field":"system.cpu.total.pct","statistic":"avg"} ], "bucket_size":"1m", "formula":"result" }, "conditions": { "items": [ {"operator":"gte","values":["0.9"],"minimum_period_match":3,"priority":"high"} ] }, "notification_config": { "enabled": true, "title":"{{ .rule_name }} triggered", "message":"CPU > 90% for {{ .condition_params[0].threshold }} ({{ .priority }})" } } ``` Loading and evaluating this rule in Go: ```go package main import ( "encoding/json" "fmt" "io/ioutil" "infini.sh/console/model/alerting" alerting2 "infini.sh/console/service/alerting" _ "infini.sh/console/service/alerting/elasticsearch" ) func main() { data, err := ioutil.ReadFile("rule.json") if err != nil { panic(err) } var rule alerting.Rule if err := json.Unmarshal(data, &rule); err != nil { panic(err) } // Retrieve the Elasticsearch engine eng := alerting2.GetEngine("elasticsearch") // Execute condition check result, err := eng.CheckCondition(&rule) if err != nil { panic(err) } fmt.Printf("Condition triggered: %v\n", len(result.ResultItems) > 0) fmt.Printf("Result items count: %d\n", len(result.ResultItems)) } ``` ### Triggering a Full Alerting Task For production deployments, rules run as scheduled tasks. The engine provides `GenerateTask` to create executable task functions: ```go eng := alerting2.GetEngine("elasticsearch") // Generate a task function for the rule task := eng.GenerateTask(&rule) // Execute with context for timeout control ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() task(ctx) // Performs full lifecycle: Do() → creates Alert, sends notifications, updates KV state ``` This approach handles the complete alerting lifecycle including throttling, escalation, and recovery detection defined in [`service/alerting/elasticsearch/engine.go`](https://github.com/infinilabs/console/blob/main/service/alerting/elasticsearch/engine.go). ## Summary - **Rule Structure**: The `Rule` struct in [`model/alerting/rule.go`](https://github.com/infinilabs/console/blob/main/model/alerting/rule.go) defines a comprehensive data model for alerting rules, encompassing resources, metrics, conditions, schedules, and notification configurations. - **Condition Types**: Conditions support both standard numeric thresholds (`ConditionItem` with operators like `gte`, `lt`) and specialized bucket-diff conditions for detecting content or volume changes across time buckets. - **Evaluation Engine**: The Elasticsearch engine in [`service/alerting/elasticsearch/engine.go`](https://github.com/infinilabs/console/blob/main/service/alerting/elasticsearch/engine.go) evaluates rules by preparing metric data via `GetTargetMetricData`, then processing conditions through `CheckBucketCondition` or `CheckCondition` depending on rule type. - **Priority and Throttling**: Conditions are evaluated by priority weight, with high-priority conditions short-circuiting lower ones. The `Do` method manages alert lifecycle, throttling, escalation, and recovery notifications. - **Expression Evaluation**: Both metric formulas and condition expressions use the `govaluate` library, with metric results stored under the `"result"` key and condition expressions generated via `ConditionItem.GenerateConditionExpression`. ## Frequently Asked Questions ### How does the alerting engine handle consecutive period requirements? The engine tracks consecutive matches using a `triggerCount` variable during condition evaluation in `CheckCondition`. When evaluating each data point, if the condition expression evaluates to true, the counter increments; otherwise, it resets to zero. The condition only fires when `triggerCount` reaches the `MinimumPeriodMatch` value defined in the `ConditionItem` struct, preventing alerts from single-point anomalies. ### What is the difference between standard conditions and bucket conditions? Standard conditions evaluate numeric thresholds against calculated metric values (like CPU > 90%), processed by the `CheckCondition` method. Bucket conditions, handled by `CheckBucketCondition`, monitor state changes across Elasticsearch buckets—such as content changes (`ContentChangeState`) or document count variations (`DocCount`)—requiring consecutive bucket matches to trigger. Bucket conditions are stored in the optional `BucketConditions` field of the `Rule` struct. ### How are condition priorities enforced during evaluation? Conditions are sorted by priority weight before evaluation begins in `CheckCondition`. The engine uses `sort.Slice` with `PriorityWeights` to arrange `ConditionItem` instances from highest to lowest priority. During the evaluation loop, once any condition triggers for a specific data series, the engine breaks out of the condition loop immediately, ensuring lower-priority conditions never execute for that series when a higher-priority one has already fired. ### Where does the alerting engine store runtime state for throttling? Runtime state fields including `LastNotificationTime`, `LastEscalationTime`, and `LastTermStartTime` are defined directly in the `Rule` struct in [`model/alerting/rule.go`](https://github.com/infinilabs/console/blob/main/model/alerting/rule.go). The `Do` method in [`service/alerting/elasticsearch/engine.go`](https://github.com/infinilabs/console/blob/main/service/alerting/elasticsearch/engine.go) reads and updates these timestamps to enforce notification throttling, manage escalation timeouts, and track recovery periods. These fields persist alongside the rule configuration, enabling stateful alerting across scheduled execution cycles.