How Easegress Performs Hot Updates Without Connection Interruption

Easegress achieves zero-downtime hot updates by forking a child process that inherits listening sockets via SIGUSR2, sharing listeners through the gracenet library, and transferring object state through an Inherit mechanism.

The open-source Easegress API gateway (megaease/easegress) implements hot updates without dropping in-flight connections through a sophisticated three-layer architecture. This approach allows administrators to reload configuration, update pipeline filters, or upgrade the binary itself while maintaining active client connections. The implementation spans process management, socket inheritance, and object lifecycle management.

The Three Pillars of Zero-Downtime Hot Updates

Graceful Update Daemon and Signal Handling

The graceupdate package in pkg/graceupdate/graceupdate.go orchestrates the hot update lifecycle. When the Easegress process receives SIGUSR2, it triggers a coordinated handoff between parent and child processes.

The signal handler, implemented in NotifySigUsr2, performs the following sequence:

1. Stops parent services via the closeCls callback
2. Forks a child process using Global.StartProcess() which inherits file descriptors
3. Waits for the child to signal readiness
4. Terminates the parent once all existing connections drain

In cmd/server.go, the server entry point wires these callbacks:

closeCls := func() {
    wg := &sync.WaitGroup{}
    wg.Add(2)
    apiServer.Close(wg)
    cls.CloseServer(wg)
    wg.Wait()
}
restartCls := func() {
    cls.StartServer()
    apiServer = api.MustNewServer(opt, cls, super, profile)
}
graceupdate.NotifySigUsr2(closeCls, restartCls)

Socket Inheritance with gracenet

To prevent connection interruption during the handoff, Easegress uses gracenet.Net (the gracenet library) to wrap network listeners. This allows the child process to inherit the listening socket file descriptors without rebinding to the address.

In pkg/object/httpserver/runtime.go, the HTTP server initialization uses gnet.Listen:

listener, err := gnet.Listen("tcp", fmt.Sprintf("%s:%d", r.spec.Address, r.spec.Port))
limitListener := limitlistener.NewLimitListener(listener, r.spec.MaxConnections)

Because the socket is inherited rather than closed and rebound, existing TCP connections remain attached to the parent process while the child immediately begins accepting new connections on the same port. This mechanism applies to HTTP/HTTPS servers, gRPC, and WASM runtimes.

Object-Level State Transfer via Inherit

Beyond socket inheritance, Easegress preserves runtime state through an object inheritance system. When a new configuration generation is loaded, the supervisor invokes InheritWithRecovery in pkg/supervisor/object.go to transfer state from the old object instance to the new one.

The inheritance contract requires objects to implement an Inherit method:

func (e *ObjectEntity) InheritWithRecovery(previousEntity *ObjectEntity, muxMapper context.MuxMapper) {
    // recover from panics to avoid crashing the whole process
    switch instance := e.Instance().(type) {
    case Controller:
        instance.Inherit(e.Spec(), previousEntity.Instance())
    case TrafficObject:
        instance.Inherit(e.Spec(), previousEntity.Instance(), muxMapper)
    }
    e.generation++
}

Concrete implementations, such as the Pipeline filter in pkg/object/pipeline/pipeline.go, use this mechanism to preserve counters, connection pools, cached TLS session tickets, and other runtime data. This ensures that hot updates maintain not just connectivity but also application-specific state.

Triggering a Hot Update in Production

To initiate a zero-downtime update, send SIGUSR2 to the running Easegress process:

# Send SIGUSR2 to the running Easegress process (PID obtained from pidfile)

kill -USR2 $(cat /var/run/easegress.pid)

Upon receiving the signal, the parent process:

1. Stops accepting new connections (handled by closeCls)
2. Forks the child process with inherited sockets
3. Continues serving existing in-flight requests
4. Exits once the child confirms readiness and all connections drain

The child process starts immediately, binding to the inherited sockets and invoking restartCls to initialize the new server instance.

How Listeners Are Shared Between Processes

The socket sharing mechanism relies on the gracenet package's ability to pass file descriptors through exec.Command with extra files. When Global.StartProcess() forks the child, it includes the listening sockets in the file descriptor table.

In pkg/object/httpserver/runtime.go, the runtime creates listeners using the gracenet wrapper:

listener, err := gnet.Listen("tcp", fmt.Sprintf("%s:%d", r.spec.Address, r.spec.Port))

This listener wraps the underlying net.TCPListener but preserves the file descriptor in a way that survives process replacement. The child process receives these descriptors and reconstructs the listeners without calling bind() again, avoiding the "address already in use" error and ensuring zero connection interruption.

Preserving Runtime State Across Updates

While socket inheritance handles network connectivity, the Inherit mechanism handles application state. When the supervisor creates a new generation of an object, it checks if the previous generation exists and calls InheritWithRecovery.

For example, in pkg/supervisor/object.go:

func (e *ObjectEntity) InheritWithRecovery(previousEntity *ObjectEntity, muxMapper context.MuxMapper) {
    // recover from panics to avoid crashing the whole process
    switch instance := e.Instance().(type) {
    case Controller:
        instance.Inherit(e.Spec(), previousEntity.Instance())
    case TrafficObject:
        instance.Inherit(e.Spec(), previousEntity.Instance(), muxMapper)
    }
    e.generation++
}

Each object type implements its own Inherit method to copy relevant state. For instance, a Pipeline might transfer request counters, while an HTTPServer might transfer TLS session cache. This ensures that after a hot update, the new process continues exactly where the old one left off, maintaining not just connections but also runtime metrics and cached data.

Summary

Easegress performs hot updates without connection interruption through a coordinated three-layer architecture:

• Process-level handoff: SIGUSR2 triggers a fork where the child inherits listening sockets via gracenet, allowing the parent to drain existing connections while the child accepts new ones.
• Socket inheritance: The gracenet.Net wrapper in pkg/object/httpserver/runtime.go preserves file descriptors across process boundaries, preventing "address already in use" errors and ensuring zero-downtime listener transfer.
• State preservation: The Inherit mechanism in pkg/supervisor/object.go transfers runtime data (counters, caches, TLS sessions) from parent to child objects, maintaining application continuity beyond just TCP connections.

Frequently Asked Questions

What signal triggers a hot update in Easegress?

Easegress uses SIGUSR2 to initiate hot updates. When the process receives this signal, the graceupdate package handles the fork and socket inheritance sequence. This signal was chosen because it is reserved for user-defined actions and does not interfere with standard process management signals like SIGTERM or SIGINT.

Does hot update support TLS session resumption?

Yes, TLS session resumption is preserved during hot updates through the Inherit mechanism. HTTPServer objects and TLS handlers implement the Inherit method to transfer session ticket caches and TLS state from the parent process to the child. This ensures that clients can resume TLS sessions without performing full handshakes after a hot update completes.

How long does the parent process remain active during a hot update?

The parent process remains active until all in-flight connections close naturally and the child process signals readiness. There is no fixed timeout enforced by the graceupdate mechanism itself; the parent waits for its http.Server to drain existing requests via the Close method callbacks defined in cmd/server.go. Once the child confirms it has started successfully, the parent exits cleanly.

Can custom filters preserve state during hot updates?

Yes, custom filters can preserve state by implementing the Inherit method defined in the object interface. When a filter implements this method, the supervisor automatically invokes it during InheritWithRecovery in pkg/supervisor/object.go, passing the previous instance as a parameter. Filters can then copy counters, caches, connection pools, or any runtime data needed to maintain continuity across the update.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}