# Harbor Job Service: Architecture, Implementation, and Background Task Management

> Discover Harbor Job Service architecture and implementation. Learn how this background processing engine manages asynchronous tasks for the Harbor container registry platform effectively.

- Repository: [Harbor/harbor](https://github.com/goharbor/harbor)
- Tags: architecture
- Published: 2026-04-09

---

**Harbor Job Service is a dedicated background-processing engine that runs asynchronous tasks for the Harbor container registry platform using a reliable job queue built on Go's `gocraft/work` library.**

The Job Service powers critical Harbor operations like image replication, garbage collection, and vulnerability scanning without blocking the main request flow. As implemented in the `goharbor/harbor` repository, this component provides a scalable, fault-tolerant system for managing long-running operations through a RESTful API and extensible worker pool architecture.

## Architecture and Core Components

The Harbor Job Service follows a modular design split between API exposure and task execution. According to the architecture documentation in [`src/jobservice/README.md`](https://github.com/goharbor/harbor/blob/main/src/jobservice/README.md), the system separates concerns into distinct components that communicate via Redis.

### Controller and Worker Pool Design

The service architecture consists of two primary runtime elements:

- **Controller**: Exposes the HTTP REST API at `/api/v1/jobs` for job submission and lifecycle management
- **Worker Pool**: Executes the actual job implementations concurrently

Both components run within the same binary by default, as seen in [`src/jobservice/main.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/main.go) (lines 44-99), where the bootstrap sequence initializes the context, logger, and starts both the controller and worker routines. However, the architecture supports independent scaling behind a load balancer, allowing horizontal scaling of workers separate from API endpoints.

### Persistent Backend and Data Layer

All job metadata persists through a **Redis-backed data store**, enabling multiple Job Service nodes to share the same queue state. The `Pool Driver` wraps the upstream `gocraft/work` library functions, while the `Persistent Driver` provides the Redis implementation that allows distributed nodes to coordinate job processing.

## Key Features and Execution Modes

The Harbor Job Service provides three distinct execution modes for background tasks, configured through the job metadata when submitting via the REST API.

### Reliable Job Queue and Execution Types

The service supports **Generic** (run once immediately), **Scheduled** (run after a delay), and **Periodic** (cron-like recurring) execution patterns. Jobs survive process crashes and support configurable retry logic through the `MaxFails()` method in the job interface.

The **Scheduler** component (located in `src/jobservice/period/`) handles cron-style periodic jobs, while the **Job Launcher** enqueues non-periodic tasks into the work queue.

### Job Lifecycle Management

Harbor Job Service provides granular control over running jobs through:

- **Stop/Cancel operations**: Immediate termination signals
- **Retry mechanisms**: Automatic re-enqueue on failure up to the `MaxFails` limit
- **Status hooks**: Callbacks for state transitions
- **Progress tracking**: Real-time check-ins via the job context

The **Stats Manager** tracks health metrics, worker pool statistics, and job execution data, while the **Logger** component (in `src/jobservice/logger/`) supports pluggable backends including STDOUT, file system, and database storage with configurable sweep policies.

## Implementing Custom Jobs in Harbor

All custom jobs must implement the interface defined in [`src/jobservice/job/interface.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/job/interface.go). The repository includes a concrete example in [`src/jobservice/job/impl/sample/job.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/job/impl/sample/job.go) demonstrating proper implementation patterns.

```go
type DemoJob struct{}

// MaxFails: allow up to 3 retries
func (dj *DemoJob) MaxFails() uint { return 3 }

// MaxCurrency: only one instance can run concurrently
func (dj *DemoJob) MaxCurrency() uint { return 1 }

func (dj *DemoJob) ShouldRetry() bool { return true }

func (dj *DemoJob) Validate(params job.Parameters) error {
    if len(params) == 0 {
        return errors.New("parameters required")
    }
    return nil
}

// Run contains the business logic
func (dj *DemoJob) Run(ctx job.Context, params job.Parameters) error {
    logger := ctx.GetLogger()
    logger.Infof("DemoJob started with params: %#v", params)

    // Periodic progress updates
    ctx.Checkin("30%")
    time.Sleep(2 * time.Second)
    ctx.Checkin("60%")
    time.Sleep(2 * time.Second)
    ctx.Checkin("100%")

    // Respect stop/cancel signals
    if cmd, ok := ctx.OPCommand(); ok && cmd == opm.CtlCommandCancel {
        return errs.JobCancelledError()
    }
    return nil
}

```

Key implementation requirements include:
- **MaxCurrency()**: Controls concurrent execution limits for the job type
- **Validate()**: Ensures parameters meet requirements before execution
- **Run()**: Contains the actual business logic with access to the job context
- **Context operations**: Use `ctx.Checkin()` for progress updates and `ctx.OPCommand()` to check for cancellation requests

## REST API for Job Submission

External Harbor components (Core, UI, CLI) interact with the Job Service through the RESTful API exposed by the controller. The API endpoint is documented in [`src/jobservice/README.md`](https://github.com/goharbor/harbor/blob/main/src/jobservice/README.md) (lines 66-90).

```bash
curl -X POST https://<harbor-host>/api/v1/jobs \
  -H "Authorization: Harbor-Secret <secret>" \
  -H "Content-Type: application/json" \
  -d '{
    "job": {
      "name": "replication",
      "parameters": {
        "src_registry": "registry1",
        "dst_registry": "registry2",
        "project": "myproj"
      },
      "metadata": {
        "kind": "Generic"
      }
    }
  }'

```

The API returns HTTP `202 Accepted` with the newly created job ID, allowing clients to poll for status or receive webhook notifications through the hook system.

## Summary

- **Harbor Job Service** provides a distributed, fault-tolerant background processing system for the Harbor container registry platform.
- The architecture separates the **Controller** (HTTP API) from the **Worker Pool** (execution), connected via Redis for state persistence.
- Three execution modes support immediate, delayed, and cron-style periodic job scheduling through the `gocraft/work` library foundation.
- Jobs implement a strict interface requiring `MaxFails()`, `MaxCurrency()`, `Validate()`, and `Run()` methods, with access to contextual logging and cancellation signals.
- The service supports horizontal scaling, process-crash recovery, and pluggable logging backends (STDOUT, File, Database).
- Implementation files are located in `src/jobservice/` with the main entry point at [`src/jobservice/main.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/main.go) and interface definitions at [`src/jobservice/job/interface.go`](https://github.com/goharbor/harbor/blob/main/src/jobservice/job/interface.go).

## Frequently Asked Questions

### What types of tasks does Harbor Job Service handle?

Harbor Job Service manages asynchronous operations including image replication between registries, garbage collection of orphaned blobs, vulnerability scanning through integrated scanners, and scheduled maintenance tasks. These operations run outside the main request flow to prevent HTTP timeouts and provide reliable execution guarantees.

### How does Harbor Job Service handle job failures and retries?

Each job implementation defines its retry policy through the `MaxFails()` method, which specifies the maximum number of attempts before marking a job as failed. The `ShouldRetry()` method allows conditional retry logic, while the underlying `gocraft/work` library ensures jobs survive process restarts and are re-enqueued according to the configured policy.

### Can Harbor Job Service scale horizontally across multiple nodes?

Yes. While the controller and worker pool run in the same binary by default, the architecture supports independent scaling behind a load balancer. Multiple Job Service instances share state through the Redis persistent backend, allowing distributed worker pools to process jobs from a shared queue while maintaining exactly-once execution semantics through the pool driver implementation.

### How do custom jobs report progress and handle cancellation?

Job implementations receive a `job.Context` parameter in the `Run()` method, which provides `ctx.Checkin()` for sending progress percentages (0-100%) to the API and `ctx.OPCommand()` to check for operator commands like `opm.CtlCommandCancel`. This allows long-running jobs to respect stop signals and provide real-time status updates to the Harbor UI and API consumers.