Understanding the Agent Module for Remote Elasticsearch Cluster Management in Infini Console

The agent module acts as a secure HTTP bridge that enables the Infini Console to discover, monitor, and manage remote Elasticsearch nodes without requiring direct network access to each cluster instance.

The infinilabs/console repository includes a sophisticated agent module that simplifies remote Elasticsearch cluster management by deploying lightweight agents on each host. This module eliminates the need for SSH access or direct Elasticsearch API exposure, instead tunneling all administrative traffic through authenticated Console-to-Agent channels.

How the Agent Module Enables Remote Cluster Discovery

The foundation of remote management lies in the agent configuration model and the enrollment system that links console instances to physical Elasticsearch nodes.

Agent Configuration and Enrollment Model

In modules/agent/model/config.go, the console defines the global agent settings through model.AgentConfig. This configuration specifies whether the agent feature is enabled, the download URL for the Infini Agent binary, TLS certificates, and the HTTP endpoint details used for communication.

When an operator enrolls a node, the console generates a unique agent ID and stores it as metadata within the node settings. This creates a persistent binding between the console's logical view and the physical host running the agent.

Node Binding via GetEnrolledNodesByAgent

The function GetEnrolledNodesByAgent in modules/agent/api/elasticsearch.go (lines 54-78) queries the console's setting store for all node_settings records where metadata.labels.agent_id matches a specific instance ID. It returns a map of BindingItem structs containing the cluster ID, node UUID, and enrollment status, enabling the console to determine which nodes belong to a particular agent instance.

Proxy Architecture for Secure Remote Access

All interactions with remote nodes route through a centralized proxy layer that handles authentication, TLS termination, and connection pooling.

The ProxyAgentRequest Transport Layer

The server.ProxyAgentRequest function, implemented in plugin/managed/server/websocket_proxy.go, serves as the transport layer for all remote-agent calls. This proxy forwards HTTP requests from the console to the agent's REST endpoint (typically listening on a configurable port like 9000), handling the underlying WebSocket or HTTP connectivity without exposing the Elasticsearch cluster directly to the console's network.

Real-time Node Discovery via GetElasticsearchNodesViaAgent

To retrieve live cluster topology, the console invokes GetElasticsearchNodesViaAgent in modules/agent/api/elasticsearch.go (lines 6-21). This function constructs a util.Request targeting the agent's /elasticsearch/node/_discovery endpoint and proxies it through server.ProxyAgentRequest. The response unmarshals into elastic.DiscoveryResult, providing fresh node metadata including version, IP addresses, and service status.

Managing Node State and Health

The agent module maintains an accurate view of cluster health by merging locally cached configuration with real-time agent reports.

Aggregating Online and Offline Status with refreshNodesInfo

The refreshNodesInfo function in modules/agent/api/elasticsearch.go (lines 99-144) produces a comprehensive node inventory by combining two data sources:

• Live node information from GetElasticsearchNodesViaAgent (nodes currently reporting via their agents)
• Direct Elasticsearch client calls for nodes not reported by agents (to detect offline or disconnected instances)

This merge process annotates each node with online/offline status, cluster_id, and enrollment state, ensuring the console displays accurate health indicators even when network partitions occur.

Handling Enrollment Metadata

When refreshNodesInfo processes the discovery results, it cross-references the live data against the enrollment map from GetEnrolledNodesByAgent. This allows the UI to distinguish between discovered nodes (visible to the agent) and enrolled nodes (officially managed by the console), preventing unauthorized nodes from appearing in management views.

Remote Log Access Without SSH

The agent module exposes specialized endpoints for operational data retrieval, eliminating the need for direct filesystem access or SSH tunnels.

Listing Log Files with GetElasticLogFiles

The GetElasticLogFiles helper in modules/agent/api/elasticsearch.go (lines 32-53) sends a POST request to the agent's /elasticsearch/logs/_list endpoint. The function requires the logs_path extracted from the node's stored configuration (via getAgentByNodeID) and returns a list of available log files on the remote host.

Streaming Log Content via GetElasticLogFileContent

For log analysis, GetElasticLogFileContent (lines 56-80) proxies requests to /elasticsearch/logs/_read, accepting parameters for file_name, offset, and lines. It returns paginated log content with a has_more flag, enabling the console to stream large log files without overwhelming the network or UI.

Implementation Examples

Discovering Enrolled Nodes

// instanceID corresponds to the Agent instance stored in the console
nodes, err := agent.GetEnrolledNodesByAgent(instanceID)
if err != nil {
    log.Fatal(err)
}

for nodeID, bind := range nodes {
    fmt.Printf("Node %s belongs to cluster %s (enrolled=%t)\n",
        nodeID, bind.ClusterID, bind.Enrolled)
}

Source: GetEnrolledNodesByAgent – modules/agent/api/elasticsearch.go:54-78

Refreshing Node Information

// instanceEndpoint is the HTTP endpoint of the remote agent (e.g., "http://10.0.2.5:9000")
info, err := agent.refreshNodesInfo(instanceID, instanceEndpoint)
if err != nil {
    log.Fatalf("cannot refresh: %v", err)
}

// info.Nodes contains enriched LocalNodeInfo structs with status and metadata
for id, node := range info.Nodes {
    fmt.Printf("Node %s – status:%s enrolled:%t\n", id, node.Status, node.Enrolled)
}

Source: refreshNodesInfo – modules/agent/api/elasticsearch.go:99-144

Retrieving Remote Log Files

// Resolve the agent instance managing this specific nodeUUID
instance, err := agent.getAgentByNodeID(nodeUUID)
if err != nil {
    log.Fatal(err)
}

// Fetch available logs from the node's configured log path
logs, err := agent.GetElasticLogFiles(
    context.Background(),
    instance,
    "/var/log/elasticsearch", // logsPath from node config
)
if err != nil {
    log.Fatal(err)
}
fmt.Println("Available log files:", logs)

Source: GetElasticLogFiles – modules/agent/api/elasticsearch.go:32-53

Reading Log Content with Pagination

body := map[string]interface{}{
    "file_name": "elasticsearch.log",
    "logs_path": "/var/log/elasticsearch",
    "offset":    0,
    "lines":     200,
}

content, err := agent.GetElasticLogFileContent(context.Background(), instance, body)
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Retrieved %d lines. Has more? %v\n", 
    len(content["lines"].([]interface{})), 
    content["has_more"])

Source: GetElasticLogFileContent – modules/agent/api/elasticsearch.go:56-80

Summary

• The agent module creates a secure abstraction layer between the Infini Console and remote Elasticsearch nodes, requiring only HTTP connectivity to lightweight agents rather than direct cluster access.
• Node discovery relies on GetEnrolledNodesByAgent and refreshNodesInfo to maintain accurate mappings between agent instances and cluster topology, tracking both online and offline states.
• Proxy architecture via server.ProxyAgentRequest in plugin/managed/server/websocket_proxy.go handles all transport concerns, including TLS and authentication, for every remote operation.
• Log management functions GetElasticLogFiles and GetElasticLogFileContent expose remote filesystem access through standardized REST endpoints, eliminating SSH dependencies.
• All agent configuration persists in modules/agent/model/config.go, enabling centralized control of agent deployment parameters and security certificates.

Frequently Asked Questions

What is the primary purpose of the agent module in Infini Console?

The agent module serves as a bridge that enables the Infini Console to manage Elasticsearch nodes deployed across network-isolated environments. By installing a lightweight Infini Agent binary on each host, operators can discover nodes, monitor health, and retrieve logs through HTTP APIs without configuring VPNs, SSH keys, or direct Elasticsearch port exposure.

How does the agent module handle network security for remote clusters?

Security relies on the server.ProxyAgentRequest implementation in plugin/managed/server/websocket_proxy.go, which establishes authenticated, encrypted channels between the console and each agent. All traffic tunnels through this proxy layer, meaning the console never connects directly to Elasticsearch nodes; it only communicates with the agent endpoints, which in turn access local resources.

Can the agent module manage multiple Elasticsearch clusters simultaneously?

Yes. The GetEnrolledNodesByAgent function returns BindingItem structs that include ClusterID fields, allowing a single agent instance to discover and report nodes from multiple distinct Elasticsearch clusters. The refreshNodesInfo function aggregates these into a unified view while maintaining cluster boundaries through metadata labels.

What endpoints does the agent expose for log management?

The agent exposes two primary endpoints for log operations: /elasticsearch/logs/_list for retrieving available log files in a directory, and /elasticsearch/logs/_read for streaming file content with offset-based pagination. The console accesses these through GetElasticLogFiles and GetElasticLogFileContent, which proxy requests via the agent's HTTP interface defined in modules/agent/api/elasticsearch.go.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}