# How Harbor Job Service Manages Worker Pools: Redis-Backed Architecture

> Discover how Harbor's Job Service manages worker pools with its Redis-backed architecture. Learn about automatic registration, reflection-based ID extraction, and reliable job re-queuing for background tasks.

- Repository: [Harbor/harbor](https://github.com/goharbor/harbor)
- Tags: internals
- Published: 2026-04-09

---

**Harbor's Job Service manages worker pools using a Redis-backed architecture built on the gocraft/work library, featuring automatic pool registration via heartbeats, reflection-based pool ID extraction, and a background reaper process that re-queues jobs from dead pools to ensure reliable execution of background tasks like vulnerability scans and retention policies.**

Harbor's Job Service orchestrates critical background operations—including image scans, retention policies, and replication tasks—through a robust worker pool system implemented in the `goharbor/harbor` repository. The pool management logic resides primarily in the `jobservice/worker` package and leverages Redis for state coordination and the third-party `gocraft/work` library for job processing primitives.

## Pool Creation and Identification

When the Job Service boots in [`src/jobservice/runtime/bootstrap.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/runtime/bootstrap.go), it reads `cfg.PoolConfig.WorkerCount` (defaulting to **10**) and constructs a `basicWorker` via `cworker.NewWorker`. This function instantiates the underlying `work.WorkerPool` from the gocraft library and establishes the Redis connection pool.

The system extracts a unique **pool ID** through reflection on the internal `workerPoolID` field. In [`src/jobservice/worker/cworker/c_worker.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/worker/cworker/c_worker.go), the `GetPoolID()` method uses `reflect.ValueOf(*w.pool).FieldByName("workerPoolID")` to access this identifier:

```go
w := cworker.NewWorker(ctx, cfg.Namespace, cfg.PoolConfig.WorkerCount, redisPool, ctl)

```

This reflection-based approach allows Harbor to track individual pool instances without modifying the third-party library's internal structures.

## Heartbeat Registration and Redis State

Worker pools maintain liveness through a **heartbeat mechanism** that writes state to Redis. The `client.WorkerPoolHeartbeats()` function records a hash entry under the key `KeyWorkerPools(namespace)`, defined in [`src/jobservice/common/rds/keys.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/common/rds/keys.go).

Each heartbeat entry contains:

- **Pool ID**: The extracted worker pool identifier
- **Start time**: When the pool initialized
- **Job names**: Currently registered job types
- **Concurrency**: Maximum concurrent workers allowed
- **Last heartbeat timestamp**: Unix timestamp of the last update

This Redis-backed registry enables horizontal scaling and allows multiple Job Service instances to discover and monitor each other's worker pools.

## Dead Pool Detection and Job Reclamation

A dedicated **reaper** process ([`src/jobservice/worker/cworker/reaper.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/worker/cworker/reaper.go)) runs two background loops to ensure system resilience. First, it synchronizes out-of-date statistics by scanning the `KeyJobTrackInProgress` hash. Second, it performs **dead-pool cleanup** by comparing active pools against theRedis set.

The reaper obtains current live pool IDs via `SMEMBERS KeyWorkerPools`. For any pool ID not present in the live set, the reaper identifies stuck jobs in that pool's dedicated in-progress queues (`KeyInProgressQueue(namespace, jobType, poolID)`).

Using the `RedisLuaReenqueueScript` Lua script, the reaper atomically:

1. Moves orphaned jobs back to the normal processing queue
2. Decrements the associated job lock counters

This mechanism guarantees that jobs from crashed or network-partitioned worker pools are eventually processed by healthy instances.

## Concurrency Control and Failure Isolation

Harbor implements **fine-grained concurrency control** through Redis-backed job locks. The system uses `KeyJobLock` and `KeyJobLockInfo` keys that embed the pool ID, ensuring that concurrency limits (`work.JobOptions.MaxConcurrency`) are respected per pool rather than globally.

Each worker pool operates independently, with the reaper starting automatically when `basicWorker.Start()` executes (`w.reaper.start()` at the end of the Start method in [`c_worker.go`](https://github.com/goharbor/harbor/blob/main/c_worker.go)). This autonomous operation prevents cascading failures and ensures that background job processing remains available even when individual pools fail.

## Code Examples

### Creating and Starting a Worker Pool

```go
import (
    "github.com/goharbor/harbor/src/jobservice/env"
    "github.com/goharbor/harbor/src/jobservice/lcm"
    "github.com/goharbor/harbor/src/jobservice/worker"
    "github.com/gomodule/redigo/redis"
)

// ctx holds system context, namespace, etc.
ctx := env.NewContext()
redisPool := &redis.Pool{ /* … */ }
ctl := lcm.NewController(/* … */)

// Build a worker with 8 goroutines
myWorker := worker.NewWorker(ctx, "harbor_jobservice", 8, redisPool, ctl)
if err := myWorker.Start(); err != nil {
    // handle start-up error
}

```

### Fetching Pool Statistics

```go
stats, err := myWorker.Stats()
if err != nil {
    // handle error
}
for _, p := range stats.Pools {
    fmt.Printf("Pool %s – status: %s, concurrency: %d\n",
        p.WorkerPoolID, p.Status, p.Concurrency)
}

```

The `StatsData` struct (defined in [`src/jobservice/worker/models.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/worker/models.go)) provides visibility into pool health, including status, concurrency limits, and job counts.

## Summary

- **Pool sizing** is configurable via `cfg.PoolConfig.WorkerCount` with a default of 10 workers per pool.
- **Unique identification** relies on reflection to extract `workerPoolID` from the underlying gocraft/work pool structures.
- **Redis coordination** uses heartbeat hashes and sorted sets to track pool membership across distributed instances.
- **Automatic recovery** through the reaper process ensures jobs from dead pools are re-queued without manual intervention.
- **Concurrency isolation** is achieved through pool-specific job locks that respect individual pool limits.

## Frequently Asked Questions

### What is the default worker pool size in Harbor Job Service?

The default worker count is **10**, configurable through `cfg.PoolConfig.WorkerCount` in the service bootstrap configuration. This value determines the number of goroutines each pool uses to process background jobs concurrently.

### How does Harbor detect dead worker pools?

The **reaper** process queries Redis using `SMEMBERS KeyWorkerPools` to retrieve the set of currently registered pools. It compares these IDs against the heartbeat timestamps; pools that haven't reported within the timeout window are considered dead and targeted for cleanup.

### What happens to in-progress jobs when a worker pool crashes?

Jobs stuck in a dead pool's `KeyInProgressQueue` are automatically reclaimed by the reaper. Using the `RedisLuaReenqueueScript` Lua script, these jobs are atomically moved back to the general work queue and their lock counters are decremented, allowing healthy pools to pick them up.

### Which Redis keys are essential for worker pool management?

Harbor uses several specialized keys defined in [`src/jobservice/common/rds/keys.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/common/rds/keys.go): `KeyWorkerPools` for pool membership tracking, `KeyInProgressQueue` for per-pool job tracking, and `KeyJobTrackInProgress` for global job status monitoring. These keys enable the distributed coordination required for multi-instance Job Service deployments.