How to Configure VarMQ's Idle Worker Expiry Duration in Go

Use the WithIdleWorkerExpiryDuration() option when creating a worker to set the maximum time idle goroutines remain in the pool before automatic cleanup.

VarMQ is a high-performance job queue library for Go that manages worker goroutines through a configurable pool. Controlling how long idle workers survive in this pool is critical for balancing resource utilization against latency when you configure VarMQ's idle worker expiry duration.

Understanding Idle Worker Expiry

Purpose and Default Behavior

The idle worker expiry duration defines how long a worker goroutine may remain unused before VarMQ automatically stops and returns it to the cache. If you do not explicitly configure this setting, the pool maintains exactly one idle worker regardless of your concurrency configuration, ensuring at least one goroutine is always ready to process new jobs.

Interaction with Minimum Idle Worker Ratio

The expiry logic operates alongside the minimum idle worker ratio configured via WithMinIdleWorkerRatio(). Even when expiry is enabled, the pool never shrinks below the threshold calculated by numMinIdleWorkers(). This prevents the pool from dropping to zero idle workers when you have specified a minimum buffer.

Implementation Details

According to the VarMQ source code, the configuration value is stored in configs.idleWorkerExpiryDuration within [config.go](https://github.com/goptics/varmq/blob/main/config.go#L19-L52).

When a worker starts, the goRemoveIdleWorkers() function in [worker.go](https://github.com/goptics/varmq/blob/main/worker.go#L66-L73) initializes a ticker that fires at the configured interval:

func (w *worker[T, JobType]) goRemoveIdleWorkers() {
    interval := w.Configs.idleWorkerExpiryDuration
    if interval == 0 { return }
    ticker := time.NewTicker(interval)
    // …
}

On each tick, the worker calculates the target number of idle workers using numMinIdleWorkers(), then prunes stale nodes. The removal logic in [worker.go](https://github.com/goptics/varmq/blob/main/worker.go#L88-L97) checks each idle worker's last-used timestamp:

for _, node := range nodes[targetIdleWorkers:] {
    if node.Value.GetLastUsed().Add(interval).Before(time.Now()) && !(node.Next() == nil && node.Prev() == nil) {
        w.pool.Remove(node)
        node.Value.Stop()
        w.pool.Cache.Put(node)
    }
}

This mechanism ensures that when you decrease concurrency, the pool eventually shrinks to match the new limits after the expiry interval elapses, as verified by the test "decrease concurrency with idle worker expiry duration" in [worker_test.go](https://github.com/goptics/varmq/blob/main/worker_test.go#L36-L84).

Configuration Examples

Per-Worker Configuration

Create a worker with a 2-second idle expiry to automatically clean up goroutines during low traffic:

import (
    "time"
    "github.com/goptics/varmq"
)

func main() {
    w := varmq.NewWorker(
        func(j varmq.Job[int]) { /* job processing */ },
        200, // concurrency
        varmq.WithIdleWorkerExpiryDuration(2*time.Second),
        varmq.WithMinIdleWorkerRatio(15), // keep ~15% idle workers
    )
}

Source: [examples/idle-wokers/main.go](https://github.com/goptics/varmq/blob/main/examples/idle-wokers/main.go)

Global Default Configuration

Set a default expiry duration for all subsequently created workers:

varmq.DefaultIdleWorkerExpiryDuration(30 * time.Second)

Source: [config.go](https://github.com/goptics/varmq/blob/main/config.go#L89-L102)

Testing with Expiry

When writing tests, use short durations to verify pool shrinking behavior:

func TestIdleWorkerExpiry(t *testing.T) {
    idleDur := 100 * time.Millisecond
    w := newWorker(myJobFunc,
        varmq.WithConcurrency(10),
        varmq.WithIdleWorkerExpiryDuration(idleDur),
    )
    // After idleDur*2, the pool should shrink to a single worker
}

Source: [worker_test.go](https://github.com/goptics/varmq/blob/main/worker_test.go#L36-L84)

When to Configure Idle Worker Expiry

High-throughput bursts: Configure expiry to maintain a large pool during traffic spikes while allowing automatic cleanup during quiet periods.

Resource-constrained environments: Use expiry to reduce goroutine count and memory overhead when the system remains idle for predictable durations.

Summary

• Configure idle worker expiry using WithIdleWorkerExpiryDuration() when creating workers, or set a global default with DefaultIdleWorkerExpiryDuration().
• The default behavior keeps exactly one idle worker regardless of concurrency settings when expiry is not configured.
• Expiry logic resides in worker.go within goRemoveIdleWorkers() and runs on a ticker at the specified interval.
• The pool never shrinks below the minimum calculated by numMinIdleWorkers(), which respects your WithMinIdleWorkerRatio() setting.
• Workers are only removed after their last-used timestamp plus the expiry interval exceeds the current time.

Frequently Asked Questions

What happens if I don't configure an idle worker expiry duration?

If you omit this configuration, VarMQ maintains exactly one idle worker in the pool indefinitely, regardless of your concurrency level. This ensures low latency for the next incoming job but prevents automatic resource reclamation during idle periods.

How does idle worker expiry interact with concurrency changes?

When you decrease concurrency, the pool may temporarily contain more workers than the new limit allows. After the configured expiry duration elapses, the excess idle workers are automatically pruned until the pool size matches the minimum idle target, typically leaving one worker ready.

Can I change the expiry duration after creating a worker?

No, the idle worker expiry duration is immutable after worker creation. It is stored in configs.idleWorkerExpiryDuration during initialization and used to start the cleanup ticker in goRemoveIdleWorkers(). To use a different duration, you must create a new worker instance.

Why does my pool still retain workers when expiry is enabled?

The pool respects the minimum idle worker ratio configured via WithMinIdleWorkerRatio(). Even with expiry enabled, VarMQ calculates the absolute minimum using numMinIdleWorkers() and never removes workers below this threshold, ensuring you maintain a buffer for sudden traffic spikes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}