How Easegress Handles Persistent Storage for Its Configuration: etcd Integration Guide

Easegress persists all runtime configuration in a distributed key‑value store (etcd), using the /config/objects/ key prefix for object storage, continuous watch-based synchronization, and optional local backup files.

The easegress-io/easegress project implements a robust persistence layer that treats etcd as the single source of truth for cluster configuration. When the server starts, it loads existing configuration from etcd (or from a local YAML file when --config-file is specified), then establishes a long-running watch on the configuration prefix to apply live updates. This architecture ensures consistency across distributed instances while supporting both embedded and external etcd deployments.

Understanding Easegress Configuration Storage Architecture

Easegress stores every configuration object—whether HTTP servers, pipelines, filters, or system controllers—as JSON values under a well-defined etcd key hierarchy. The storage mechanism relies on three tightly-coupled components defined in the source tree:

• pkg/cluster/layout.go – Defines the etcd key schema via ConfigObjectKey() and ConfigObjectPrefix().
• pkg/supervisor/supervisor.go – Handles initial writes of system controller specifications to etcd via syncSystemControllerInCluster().
• pkg/supervisor/object.go – Implements the ObjectRegistry that watches etcd changes and maintains the in-process cache.

The etcd Key Layout

In pkg/cluster/layout.go, the Layout struct provides deterministic key generation for all configuration objects:

const (
    configObjectPrefix = "/config/objects/"
    configObjectFormat = "/config/objects/%s"
)

func (l *Layout) ConfigObjectKey(name string) string {
    return fmt.Sprintf(configObjectFormat, name)
}

Every object persisted by Easegress occupies a key matching /config/objects/<object-name>. For example, an HTTP server named web-server stores its JSON specification at /config/objects/web-server. A separate key /config/version tracks the overall configuration version for cluster-wide consistency checks.

How Easegress Writes and Syncs Configuration Data

The persistence flow involves two distinct phases: initial seeding of configuration at startup, and continuous bidirectional synchronization during runtime.

Initial Configuration Loading

When the Easegress server initializes, the option package (pkg/option/option.go) processes command-line flags. If --config-file (or -f) is provided, the server loads the YAML specification locally and bypasses other flags:

if opt.ConfigFile != "" {
    opt.viper.SetConfigFile(opt.ConfigFile)
    opt.viper.SetConfigType("yaml")
    err := opt.viper.ReadInConfig()
    // ... unmarshals into opt struct
}

For initial object configurations specified via --initial-object-config-files, the supervisor (pkg/supervisor/supervisor.go) reads each file, parses it into a Spec, and writes it to etcd only if the key does not already exist:

func (s *Supervisor) syncSystemControllerInCluster(spec *Spec) {
    value, err := s.cls.Get(s.cls.Layout().ConfigObjectKey(spec.Name()))
    if err != nil { panic(err) }
    if value != nil { return } // already present
    
    err = s.cls.Put(
        s.cls.Layout().ConfigObjectKey(spec.Name()), 
        spec.JSONConfig()
    )
    if err != nil { panic(err) }
}

This ensures that etcd remains the authoritative source while preventing accidental overwrites of existing cluster state.

Continuous Synchronization with ObjectRegistry

Once initialized, the ObjectRegistry (pkg/supervisor/object.go) maintains consistency between etcd and the running system. It creates a syncer that watches the /config/objects/ prefix:

syncer := cluster.NewSyncer(cls, layout.ConfigObjectPrefix())
syncChan, err := syncer.SyncPrefix(prefix)

A background goroutine (run()) consumes events from syncChan. For each batch of changes, it:

1. Strips the prefix from etcd keys to isolate object names.
2. Applies configuration via applyConfig(), creating, updating, or deleting ObjectEntity instances in memory.
3. Persists a local backup via storeConfigInLocal(), writing the entire configuration map to <HOME>/running_objects.bak.json.

func (or *ObjectRegistry) run() {
    for {
        select {
        case kv := <-or.configSyncChan:
            config := make(map[string]string)
            for k, v := range kv {
                k = strings.TrimPrefix(k, or.configPrefix)
                config[k] = v
            }
            or.applyConfig(config)
            or.storeConfigInLocal(config)
        }
    }
}

Because the syncer utilizes etcd's native watch API, any external modification—whether from another Easegress node or a direct etcdctl command—propagates to all connected instances within milliseconds.

Embedded vs External etcd Modes

Easegress supports two deployment models for the etcd backend, controlled by flags in pkg/option/option.go:

• Embedded etcd (default): The supervisor spawns an etcd instance inside the Easegress process, storing data in the local filesystem. This mode requires no external dependencies and is suitable for single-node or development deployments.
• External etcd: When --use-standalone-etcd is set to true, Easegress connects to an existing etcd cluster using the endpoints specified in the configuration. This mode is recommended for production environments requiring high availability and horizontal scaling.

// pkg/option/option.go
UseStandaloneEtcd bool // flag --use-standalone-etcd

The persistence logic remains identical in both modes; only the etcd client initialization differs. In embedded mode, the data directory is managed by Easegress itself, while external mode delegates storage lifecycle management to the cluster administrator.

Practical Example: Interacting with Easegress Configuration Storage

To demonstrate the persistence mechanism in action, consider a scenario where you start Easegress with an embedded etcd instance and then modify configuration via etcdctl.

First, start the server with a configuration file:

# Start Easegress (embedded etcd) with a config file

easegress-server -c my-config.yaml

Next, inspect the current configuration keys in etcd:

etcdctl get --prefix /config/objects/

Add a new HTTP server object directly to the persistent store:

etcdctl put /config/objects/my-http-server \
'{"kind":"HTTPServer","name":"my-http-server","spec":{"port":8080}}'

The running Easegress instance detects the change via its watch on /config/objects/ and immediately instantiates the new HTTP server. You can verify the object is active:

easegress-client get object my-http-server

If you restart the Easegress process, it reloads the configuration from etcd, ensuring the my-http-server object persists across restarts. Additionally, the local backup file <HOME>/running_objects.bak.json provides a fallback in case of etcd connectivity issues.

Summary

Easegress handles persistent storage for its configuration through a robust etcd-backed architecture that ensures consistency and durability across distributed deployments:

• Etcd as source of truth: All configuration objects reside under the /config/objects/ prefix in etcd, with keys generated by ConfigObjectKey() in pkg/cluster/layout.go.
• Dual-phase initialization: The supervisor seeds etcd with initial configurations from files at startup (syncSystemControllerInCluster), then transitions to continuous watch-based synchronization.
• Real-time synchronization: The ObjectRegistry watches etcd changes via SyncPrefix, applying updates to in-memory objects and maintaining a local backup at running_objects.bak.json.
• Flexible deployment: Supports both embedded etcd for simplicity and external etcd clusters for high availability, controlled by the --use-standalone-etcd flag.

Frequently Asked Questions

How does Easegress ensure configuration persistence across node restarts?

Easegress stores all configuration in etcd, which persists data to disk independently of the Easegress process. When a node restarts, the supervisor loads the configuration from etcd using the ConfigObjectPrefix() keys, reconstructing the entire object state. The local running_objects.bak.json file serves as a secondary cache but does not replace etcd as the primary persistence layer.

Can Easegress run without an external etcd cluster?

Yes, Easegress can run with an embedded etcd instance activated by default when --use-standalone-etcd is false. In this mode, the supervisor spawns an etcd server within the same process, storing data in the local filesystem. This configuration is suitable for development or single-node deployments but lacks the high availability characteristics of a multi-node external etcd cluster.

What happens when configuration is modified directly in etcd?

When you modify a configuration key under /config/objects/ using etcdctl or another etcd client, the change triggers Easegress's watch mechanism. The ObjectRegistry receives the event through configSyncChan, parses the JSON configuration, and calls applyConfig to update or create the corresponding in-memory object. This ensures that all Easegress instances in the cluster converge to the new configuration state within milliseconds.

Where does Easegress store local configuration backups?

While etcd remains the authoritative store, Easegress maintains a local backup file named running_objects.bak.json in the home directory (or configured data directory). The storeConfigInLocal function in pkg/supervisor/object.go writes the entire configuration map to this file after every successful synchronization from etcd. This backup allows the system to recover basic configuration information even if etcd becomes temporarily unavailable, though the primary persistence mechanism remains the distributed key-value store.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}