How Harbor Job Service Manages Worker Pools: Redis-Backed Architecture

Harbor's Job Service manages worker pools using a Redis-backed architecture built on the gocraft/work library, featuring automatic pool registration via heartbeats, reflection-based pool ID extraction, and a background reaper process that re-queues jobs from dead pools to ensure reliable execution of background tasks like vulnerability scans and retention policies.

Harbor's Job Service orchestrates critical background operations—including image scans, retention policies, and replication tasks—through a robust worker pool system implemented in the goharbor/harbor repository. The pool management logic resides primarily in the jobservice/worker package and leverages Redis for state coordination and the third-party gocraft/work library for job processing primitives.

Pool Creation and Identification

When the Job Service boots in src/jobservice/runtime/bootstrap.go, it reads cfg.PoolConfig.WorkerCount (defaulting to 10) and constructs a basicWorker via cworker.NewWorker. This function instantiates the underlying work.WorkerPool from the gocraft library and establishes the Redis connection pool.

The system extracts a unique pool ID through reflection on the internal workerPoolID field. In src/jobservice/worker/cworker/c_worker.go, the GetPoolID() method uses reflect.ValueOf(*w.pool).FieldByName("workerPoolID") to access this identifier:

w := cworker.NewWorker(ctx, cfg.Namespace, cfg.PoolConfig.WorkerCount, redisPool, ctl)

This reflection-based approach allows Harbor to track individual pool instances without modifying the third-party library's internal structures.

Heartbeat Registration and Redis State

Worker pools maintain liveness through a heartbeat mechanism that writes state to Redis. The client.WorkerPoolHeartbeats() function records a hash entry under the key KeyWorkerPools(namespace), defined in src/jobservice/common/rds/keys.go.

Each heartbeat entry contains:

• Pool ID: The extracted worker pool identifier
• Start time: When the pool initialized
• Job names: Currently registered job types
• Concurrency: Maximum concurrent workers allowed
• Last heartbeat timestamp: Unix timestamp of the last update

This Redis-backed registry enables horizontal scaling and allows multiple Job Service instances to discover and monitor each other's worker pools.

Dead Pool Detection and Job Reclamation

A dedicated reaper process (src/jobservice/worker/cworker/reaper.go) runs two background loops to ensure system resilience. First, it synchronizes out-of-date statistics by scanning the KeyJobTrackInProgress hash. Second, it performs dead-pool cleanup by comparing active pools against theRedis set.

The reaper obtains current live pool IDs via SMEMBERS KeyWorkerPools. For any pool ID not present in the live set, the reaper identifies stuck jobs in that pool's dedicated in-progress queues (KeyInProgressQueue(namespace, jobType, poolID)).

Using the RedisLuaReenqueueScript Lua script, the reaper atomically:

1. Moves orphaned jobs back to the normal processing queue
2. Decrements the associated job lock counters

This mechanism guarantees that jobs from crashed or network-partitioned worker pools are eventually processed by healthy instances.

Concurrency Control and Failure Isolation

Harbor implements fine-grained concurrency control through Redis-backed job locks. The system uses KeyJobLock and KeyJobLockInfo keys that embed the pool ID, ensuring that concurrency limits (work.JobOptions.MaxConcurrency) are respected per pool rather than globally.

Each worker pool operates independently, with the reaper starting automatically when basicWorker.Start() executes (w.reaper.start() at the end of the Start method in c_worker.go). This autonomous operation prevents cascading failures and ensures that background job processing remains available even when individual pools fail.

Code Examples

Creating and Starting a Worker Pool

import (
    "github.com/goharbor/harbor/src/jobservice/env"
    "github.com/goharbor/harbor/src/jobservice/lcm"
    "github.com/goharbor/harbor/src/jobservice/worker"
    "github.com/gomodule/redigo/redis"
)

// ctx holds system context, namespace, etc.
ctx := env.NewContext()
redisPool := &redis.Pool{ /* … */ }
ctl := lcm.NewController(/* … */)

// Build a worker with 8 goroutines
myWorker := worker.NewWorker(ctx, "harbor_jobservice", 8, redisPool, ctl)
if err := myWorker.Start(); err != nil {
    // handle start-up error
}

Fetching Pool Statistics

stats, err := myWorker.Stats()
if err != nil {
    // handle error
}
for _, p := range stats.Pools {
    fmt.Printf("Pool %s – status: %s, concurrency: %d\n",
        p.WorkerPoolID, p.Status, p.Concurrency)
}

The StatsData struct (defined in src/jobservice/worker/models.go) provides visibility into pool health, including status, concurrency limits, and job counts.

Summary

• Pool sizing is configurable via cfg.PoolConfig.WorkerCount with a default of 10 workers per pool.
• Unique identification relies on reflection to extract workerPoolID from the underlying gocraft/work pool structures.
• Redis coordination uses heartbeat hashes and sorted sets to track pool membership across distributed instances.
• Automatic recovery through the reaper process ensures jobs from dead pools are re-queued without manual intervention.
• Concurrency isolation is achieved through pool-specific job locks that respect individual pool limits.

Frequently Asked Questions

What is the default worker pool size in Harbor Job Service?

The default worker count is 10, configurable through cfg.PoolConfig.WorkerCount in the service bootstrap configuration. This value determines the number of goroutines each pool uses to process background jobs concurrently.

How does Harbor detect dead worker pools?

The reaper process queries Redis using SMEMBERS KeyWorkerPools to retrieve the set of currently registered pools. It compares these IDs against the heartbeat timestamps; pools that haven't reported within the timeout window are considered dead and targeted for cleanup.

What happens to in-progress jobs when a worker pool crashes?

Jobs stuck in a dead pool's KeyInProgressQueue are automatically reclaimed by the reaper. Using the RedisLuaReenqueueScript Lua script, these jobs are atomically moved back to the general work queue and their lock counters are decremented, allowing healthy pools to pick them up.

Which Redis keys are essential for worker pool management?

Harbor uses several specialized keys defined in src/jobservice/common/rds/keys.go: KeyWorkerPools for pool membership tracking, KeyInProgressQueue for per-pool job tracking, and KeyJobTrackInProgress for global job status monitoring. These keys enable the distributed coordination required for multi-instance Job Service deployments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}