Data Model for Alerting Rules and Condition Evaluation in InfiniLabs Console

The alerting engine in InfiniLabs Console uses a JSON-driven Rule struct that combines resource targets, metric expressions, and prioritized conditions to evaluate Elasticsearch data via govaluate expressions.

The InfiniLabs Console repository (infinilabs/console) provides a comprehensive alerting subsystem designed to monitor Elasticsearch clusters. At its core lies a structured data model for alerting rules that supports both numeric thresholds and complex bucket-diff conditions. This article examines the Rule struct definition, condition evaluation logic, and the engine implementation that processes these rules against time-series data.

Alerting Rule Data Model

The Rule Struct

The foundation of the alerting system is the Rule struct defined in model/alerting/rule.go. This structure encapsulates all configuration necessary to define, schedule, and execute a monitoring rule:

• Identification and Metadata: ID, Created, Updated, Name, and Enabled fields provide basic rule management.
• Resource Targeting: The Resource field (defined in model/alerting/resource.go) specifies the Elasticsearch indices, time field, and raw filters to query.
• Metric Configuration: The Metrics field (from model/alerting/metric.go) defines aggregation functions, bucket sizes, and mathematical formulas.
• Condition Logic: Conditions and optional BucketConditions (from model/alerting/condition.go) establish the logical thresholds that trigger alerts.
• Notification Settings: Channels, NotificationConfig, and RecoveryNotificationConfig control webhook, email, and escalation behaviors.
• Scheduling: The Schedule field (from model/alerting/schedule.go) provides cron-like execution timing.
• Runtime State: Fields like LastNotificationTime, LastEscalationTime, and LastTermStartTime manage throttling and stateful tracking.

The struct also provides Rule.GetOrInitExpression, which generates a full evaluable expression combining metric formulas with condition expressions.

Condition Definitions

Conditions are defined in model/alerting/condition.go through two primary structures:

Condition acts as a container with:

• Operator: Currently supports "or" logic between items
• Items: A slice of ConditionItem structs

ConditionItem represents individual threshold checks with:

• Operator: Comparison operators (equals, gte, lt, gt, lte, neq)
• Values: Threshold values as strings
• MinimumPeriodMatch: Number of consecutive data points that must satisfy the condition
• Priority: Severity level (affects evaluation order)
• Type and BucketCount: Special fields for bucket-diff conditions

The ConditionItem.GenerateConditionExpression method produces evaluable expressions using the placeholder variable result. For example, a "gte" operator generates:

conditionExpression = fmt.Sprintf("result >= %v", cond.Values[0])

Metric Expressions

The Metric struct in model/alerting/metric.go defines how to extract and compute values from Elasticsearch:

• Items: List of MetricItem structs specifying fields and aggregations (max, avg, percentile, etc.)
• BucketSize: Time interval for aggregation
• Formula: Mathematical expression combining metric items (e.g., "a + b" or "result")

The Metric.GenerateExpression method constructs the evaluable formula used by the engine.

How Conditions Are Evaluated in the Alerting Engine

The evaluation logic resides in service/alerting/elasticsearch/engine.go. The engine processes rules through a pipeline that prepares metric data, evaluates conditions by priority, and manages stateful notifications.

Data Preparation and Metric Calculation

The engine.GetTargetMetricData method orchestrates the initial data retrieval:

1. Query Generation: Constructs Elasticsearch queries using GenerateQuery based on the rule's resource and metric definitions.
2. Data Retrieval: Executes the query against the target cluster.
3. Formula Evaluation: For rules with multiple metric items, evaluates the user-supplied formula using govaluate.NewEvaluableExpression. The result is a slice of insight.MetricData where computed values are stored under the key "result".

This preparation ensures that subsequent condition evaluation works against normalized result values regardless of the underlying metric complexity.

Bucket Condition Evaluation

For rules requiring stateful bucket analysis (such as detecting content changes or document count anomalies), the engine uses CheckBucketCondition (lines 600-740 in engine.go):

1. Timestamp Collation: Gathers all unique bucket timestamps across metric groups.
2. Diff State Calculation: Computes either ContentChangeState (for content-based rules) or DocCount (for volume-based rules) for each timestamp.
3. Expression Evaluation: Evaluates bucket-condition expressions using govaluate against the computed diff values.
4. Consecutive Period Matching: Tracks how many consecutive buckets satisfy the condition. When the count reaches MinimumPeriodMatch, the condition triggers.

This approach supports complex scenarios like "alert if index content changes for 3 consecutive minutes" or "alert if document count drops for 2 consecutive buckets."

Standard Numeric Condition Checking

When no bucket conditions exist, CheckCondition (lines 870-990 in engine.go) handles numeric threshold evaluation:

Priority-Based Ordering: Conditions are sorted by priority weight to ensure critical alerts evaluate first:

sort.Slice(rule.Conditions.Items, func(i, j int) bool {
    return alerting.PriorityWeights[rule.Conditions.Items[i].Priority] >
           alerting.PriorityWeights[rule.Conditions.Items[j].Priority]
})

Per-Series Evaluation: For each metric data series in targetMetricData:

• Determines the shortest series length and identifies the dataKey containing result values.
• Iterates through each data point index i.
• Builds a relationValues map containing raw values of all metric items for that timestamp.
• Evaluates the metric formula using these values to produce valueExpressionResult.
• Applies the condition expression (generated from ConditionItem) to valueExpressionResult using govaluate.
• Maintains a triggerCount for consecutive matches. When triggerCount reaches cond.MinimumPeriodMatch, the condition is marked triggered and a ConditionResultItem is recorded.
• Stops checking further conditions for that series once a condition fires (breaks the LoopCondition).

This design ensures efficient evaluation where high-priority conditions short-circuit lower-priority checks, and multi-period requirements prevent flapping alerts from single-point anomalies.

Notification and Escalation Handling

After condition evaluation completes, the Do method (lines 530-730 in engine.go) manages the alert lifecycle:

• Alert Creation: Creates or updates Alert records based on ConditionResult.
• Throttling: Checks LastNotificationTime against notification configuration to prevent spam.
• Escalation: Tracks LastEscalationTime to escalate unacknowledged alerts after configured periods.
• Recovery Detection: When conditions no longer trigger, sends recovery notifications using RecoveryNotificationConfig.
• Channel Dispatch: Routes messages through configured channels (webhook, email) using template resolution from service/alerting/common.

Practical Implementation Examples

Defining and Loading a Rule

The following JSON structure represents a complete alerting rule configuration targeting high CPU usage:

{
  "name": "CPU usage high",
  "enabled": true,
  "resource": {
    "id": "es-cluster-1",
    "objects": ["metrics*"],
    "time_field": "@timestamp"
  },
  "metrics": {
    "items": [
      {"name":"cpu","field":"system.cpu.total.pct","statistic":"avg"}
    ],
    "bucket_size":"1m",
    "formula":"result"
  },
  "conditions": {
    "items": [
      {"operator":"gte","values":["0.9"],"minimum_period_match":3,"priority":"high"}
    ]
  },
  "notification_config": {
    "enabled": true,
    "title":"{{ .rule_name }} triggered",
    "message":"CPU > 90% for {{ .condition_params[0].threshold }} ({{ .priority }})"
  }
}

Loading and evaluating this rule in Go:

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    
    "infini.sh/console/model/alerting"
    alerting2 "infini.sh/console/service/alerting"
    _ "infini.sh/console/service/alerting/elasticsearch"
)

func main() {
    data, err := ioutil.ReadFile("rule.json")
    if err != nil {
        panic(err)
    }
    
    var rule alerting.Rule
    if err := json.Unmarshal(data, &rule); err != nil {
        panic(err)
    }

    // Retrieve the Elasticsearch engine
    eng := alerting2.GetEngine("elasticsearch")
    
    // Execute condition check
    result, err := eng.CheckCondition(&rule)
    if err != nil {
        panic(err)
    }
    
    fmt.Printf("Condition triggered: %v\n", len(result.ResultItems) > 0)
    fmt.Printf("Result items count: %d\n", len(result.ResultItems))
}

Triggering a Full Alerting Task

For production deployments, rules run as scheduled tasks. The engine provides GenerateTask to create executable task functions:

eng := alerting2.GetEngine("elasticsearch")

// Generate a task function for the rule
task := eng.GenerateTask(&rule)

// Execute with context for timeout control
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

task(ctx) // Performs full lifecycle: Do() → creates Alert, sends notifications, updates KV state

This approach handles the complete alerting lifecycle including throttling, escalation, and recovery detection defined in service/alerting/elasticsearch/engine.go.

Summary

• Rule Structure: The Rule struct in model/alerting/rule.go defines a comprehensive data model for alerting rules, encompassing resources, metrics, conditions, schedules, and notification configurations.

• Condition Types: Conditions support both standard numeric thresholds (ConditionItem with operators like gte, lt) and specialized bucket-diff conditions for detecting content or volume changes across time buckets.

• Evaluation Engine: The Elasticsearch engine in service/alerting/elasticsearch/engine.go evaluates rules by preparing metric data via GetTargetMetricData, then processing conditions through CheckBucketCondition or CheckCondition depending on rule type.

• Priority and Throttling: Conditions are evaluated by priority weight, with high-priority conditions short-circuiting lower ones. The Do method manages alert lifecycle, throttling, escalation, and recovery notifications.

• Expression Evaluation: Both metric formulas and condition expressions use the govaluate library, with metric results stored under the "result" key and condition expressions generated via ConditionItem.GenerateConditionExpression.

Frequently Asked Questions

How does the alerting engine handle consecutive period requirements?

The engine tracks consecutive matches using a triggerCount variable during condition evaluation in CheckCondition. When evaluating each data point, if the condition expression evaluates to true, the counter increments; otherwise, it resets to zero. The condition only fires when triggerCount reaches the MinimumPeriodMatch value defined in the ConditionItem struct, preventing alerts from single-point anomalies.

What is the difference between standard conditions and bucket conditions?

Standard conditions evaluate numeric thresholds against calculated metric values (like CPU > 90%), processed by the CheckCondition method. Bucket conditions, handled by CheckBucketCondition, monitor state changes across Elasticsearch buckets—such as content changes (ContentChangeState) or document count variations (DocCount)—requiring consecutive bucket matches to trigger. Bucket conditions are stored in the optional BucketConditions field of the Rule struct.

How are condition priorities enforced during evaluation?

Conditions are sorted by priority weight before evaluation begins in CheckCondition. The engine uses sort.Slice with PriorityWeights to arrange ConditionItem instances from highest to lowest priority. During the evaluation loop, once any condition triggers for a specific data series, the engine breaks out of the condition loop immediately, ensuring lower-priority conditions never execute for that series when a higher-priority one has already fired.

Where does the alerting engine store runtime state for throttling?

Runtime state fields including LastNotificationTime, LastEscalationTime, and LastTermStartTime are defined directly in the Rule struct in model/alerting/rule.go. The Do method in service/alerting/elasticsearch/engine.go reads and updates these timestamps to enforce notification throttling, manage escalation timeouts, and track recovery periods. These fields persist alongside the rule configuration, enabling stateful alerting across scheduled execution cycles.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}