How to Configure a Custom Backoff Strategy for Keyed Routines in Go

Use keyed.WithBackoff() to supply a per-key factory function or keyed.WithRetry() for a global static policy when constructing your Keyed manager in the aperturerobotics/util repository.

The aperturerobotics/util library provides a generic Keyed manager that orchestrates long-running goroutines by unique identifiers. When you configure a custom backoff strategy for keyed routines, you control exactly how the system responds to failures—whether applying a uniform policy globally or tailoring retry intervals to individual keys via the option helpers in keyed/keyed-opts.go.

Core Architecture

The retry mechanism relies on four primary components:

• Keyed (keyed/keyed.go): The manager that stores the per-key backoff factory (backoffFactory) and the map of active routines.
• Option (keyed/keyed-opts.go): Configuration helpers including WithRetry and WithBackoff that inject backoff behavior at construction time.
• runningRoutine (keyed/routine.go): Per-key state that holds the retry backoff instance (retryBo) and schedules restarts using time.AfterFunc.
• BackOff interface (backoff/cbackoff/backoff.go): The contract requiring NextBackOff() time.Duration, which returns the wait interval or backoff.Stop to halt retries.

When a routine exits, the execute method (lines 31–48 of routine.go) checks for a configured backoff. If present, it calls NextBackOff() to determine the retry delay; upon success, it invokes retryBo.Reset() to clear the backoff state.

Global Backoff with WithRetry

To apply a single backoff configuration to all keys, use WithRetry. This option stores a factory that returns the same cbackoff.BackOff instance for every key.

import (
    "github.com/aperturerobotics/util/backoff"
    "github.com/aperturerobotics/util/keyed"
)

// Build an exponential backoff configuration
bo := &backoff.Backoff{
    BackoffKind: backoff.BackoffKind_BackoffKind_EXPONENTIAL,
    Exponential: &backoff.Exponential{
        InitialIntervalMs: 100,      // 100 ms
        MaxIntervalMs:    5_000,    // 5 s
        MaxElapsedTimeMs: 60_000,   // stop after 1 min
    },
}

// Create a keyed manager with global retry policy
km := keyed.NewKeyed(
    myCtor,                     // func(key K) (Routine, V)
    keyed.WithRetry[MyKey, MyValue](bo),
)

As implemented in keyed-opts.go (lines 46–55), WithRetry converts the protobuf *backoff.Backoff into a concrete cbackoff.BackOff using bo.Construct(), then wraps it in a factory function that ignores the key parameter.

Per-Key Backoff with WithBackoff

For key-specific policies—such as faster retries for high-priority services—supply a factory function to WithBackoff. The factory receives the key and returns a cbackoff.BackOff (or nil to disable retries for that key).

import (
    "time"
    "github.com/aperturerobotics/util/backoff"
    "github.com/aperturerobotics/util/backoff/cbackoff"
    "github.com/aperturerobotics/util/keyed"
)

// Example key-specific exponential backoff
func makeBackoff(k ServiceKey) cbackoff.BackOff {
    // Faster retry for high-priority keys
    if k.Priority == "high" {
        cfg := &backoff.Backoff{
            BackoffKind: backoff.BackoffKind_BackoffKind_EXPONENTIAL,
            Exponential: &backoff.Exponential{
                InitialIntervalMs: 50,
                MaxIntervalMs:    500,
                MaxElapsedTimeMs: 30_000,
            },
        }
        return cfg.Construct()
    }
    // Default backoff for everything else
    return cbackoff.NewConstantBackOff(2 * time.Second)
}

// Build the manager with per-key backoff
km := keyed.NewKeyed(
    myCtor,
    keyed.WithBackoff[ServiceKey, ServiceValue](makeBackoff),
)

According to the source in keyed-opts.go (lines 58–65), WithBackoff stores the provided function directly into Keyed.backoffFactory. When newRunningRoutine (lines 53–65 of routine.go) creates a new routine instance, it invokes this factory with the specific key, assigning the result to retryBo.

Disabling Retries

To make a routine non-retryable, return nil from your WithBackoff factory for that specific key, or pass nil to WithRetry for a global disable. When retryBo is nil, the execute method in routine.go skips the retry scheduling logic and the routine exits permanently on error.

Retry Mechanics and Scheduling

The actual retry logic resides in runningRoutine.execute (routine.go, lines 31–48). After a routine finishes:

1. Success: Calls r.retryBo.Reset() to clear accumulated delay state.
2. Failure: Computes dur := r.retryBo.NextBackOff().
3. Scheduling: If dur != backoff.Stop, it schedules a restart:

r.deferRetry = time.AfterFunc(dur, func() {
    r.k.mtx.Lock()
    if r.k.ctx != nil && r.k.routines[r.key] == r && r.exited {
        r.start(r.k.ctx, r.exitedCh, true) // force restart after wait
    }
    r.k.mtx.Unlock()
})

This ensures that each key’s retry timing is isolated and that backoff state persists across restart attempts until success or until MaxElapsedTime triggers backoff.Stop.

Summary

• Global policies: Use keyed.WithRetry(protoConfig) to share one backoff strategy across all keys.
• Per-key policies: Use keyed.WithBackoff(func(k K) cbackoff.BackOff { ... }) to tailor intervals to individual keys.
• Disable retries: Return nil from the factory function to prevent automatic restart for specific keys.
• State management: The backoff instance is created per routine in newRunningRoutine and reset on success via retryBo.Reset().
• Scheduling: Retries are handled automatically by runningRoutine.execute using time.AfterFunc driven by NextBackOff().

Frequently Asked Questions

Can I use different backoff strategies for different keys?

Yes. Pass a factory function to keyed.WithBackoff() that inspects the key and returns different cbackoff.BackOff implementations. For example, return an exponential backoff for critical services and a constant backoff for background tasks.

How do I disable retries for specific keys only?

Inside your WithBackoff factory function, check the key and return nil for those you do not want to retry. The manager treats a nil backoff as a signal to exit permanently on error rather than scheduling a restart.

What happens to the backoff state when a routine succeeds?

When a routine completes without error, runningRoutine.execute calls retryBo.Reset(), which clears any accumulated delay intervals. This ensures that the next failure starts from the initial backoff interval rather than the previous capped value.

Where is the retry scheduling logic implemented?

The scheduling logic lives in keyed/routine.go within the runningRoutine.execute method (lines 31–48). This code checks for errors, queries NextBackOff(), and uses time.AfterFunc to trigger r.start() after the computed delay.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}