How the Image Prebuild System Workflow Operates in Background Agents

The image prebuild system workflow in Background Agents accelerates session start-up by caching repository environments through a three-phase pipeline—Enable & Schedule, Build Execution, and Session Start-up—that spans a TypeScript control-plane and a Python Modal data-plane.

The ColeMurray/background-agents repository implements this workflow to eliminate cold-start latency. When enabled, the system bakes a pre-built image for each repository or environment, allowing new sessions to boot from a snapshot rather than cloning and running setup scripts from scratch. This architecture delegates orchestration to the control-plane while delegating heavy compute to Modal’s sandbox infrastructure.

The Three Core Phases of the Image Prebuild Workflow

Phase 1: Enable and Schedule

Activation begins in the UI. When a user toggles pre-built images for a repository or environment, the frontend immediately triggers the first build and persists the configuration.

• UI Components: The settings interface is implemented in packages/web/src/components/settings/images-settings.tsx and image-settings.tsx.
• Data Persistence: The control-plane stores the configuration in packages/control-plane/src/db/image-builds.ts, which defines the schema for tracking build status, fingerprints, and provider image IDs.

Once enabled, the unit enters the scheduler’s pool for continuous evaluation.

Phase 2: Build Execution

The control-plane posts a build request to the Modal image builder via the build_image function located in packages/modal-infra/src/scheduler/image_builder.py. The execution follows this sequence:

1. Sandbox Creation: The builder creates a sandbox, clones the repository set on the base branch, and sequentially executes each repo’s .openinspect/setup.sh.
2. Log Streaming: Structured logs are captured in real-time via _stream_build_logs, with failure handling managed by _callback_with_retry.
3. Snapshot and Callback: On success, the sandbox snapshots its filesystem into a provider-specific artifact (Modal image, Vercel snapshot, or OpenComputer checkpoint). The result—including provider image ID, per-repo SHAs, runtime version, and build duration—is POSTed back to the control-plane via an HMAC-authenticated callback using generate_internal_token for JWT signing.

Phase 3: Session Start-up

When a new session launches, the control-plane evaluates whether a pre-built image is available.

• Fingerprint Matching: The system checks the fingerprint—a deterministic hash of the ordered repository list and base branches—against ready images in packages/control-plane/src/sandbox/lifecycle/image-selection.ts.
• Fast Boot: If a matching ready image exists, the sandbox boots from that snapshot and performs a fast git sync for any new commits.
• Fallback: If no ready image matches (disabled, building, failed, or outdated), the session falls back to the standard clone-and-setup flow, with entry points defined in packages/control-plane/src/routes/image-builds.ts.

The Scheduler and Rebuild Triggers

A Modal cron function acts as the scheduler, running every 30 minutes to maintain image freshness. The scheduler evaluates all enabled units and triggers rebuilds based on specific criteria:

• Missing Images: No ready image exists for the current fingerprint.
• Runtime Floor: The sandbox runtime version is below the current minimum (min_runtime_version).
• SHA Drift: The remote repository’s HEAD SHA differs from the recorded SHA.

To prevent overload, the scheduler caps concurrent builds at TRIGGER_CAP_PER_TICK = 8. It also performs garbage collection, marking stale images as failed and cleaning up old failed rows.

Key Concepts and Security Mechanisms

Fingerprint: A deterministic hash representing the repository list and base branches. Any alteration—adding a repo, changing order, or switching branches—generates a new fingerprint, automatically retiring the cached image.

Runtime Floor: The scheduler enforces a minimum sandbox runtime version. Images built on older runtimes are flagged for reconstruction to ensure compatibility.

Callback Security: Results from the Modal data-plane to the control-plane are authenticated via HMAC-signed JWTs generated by generate_internal_token. This prevents SSRF attacks and ensures build results originate from trusted workers.

Failure Handling: Build failures are reported through the failure callback, surfacing a "Failed" state in the UI. The scheduler automatically retries failed builds on the next tick.

Implementation Examples

Trigger a manual rebuild via the control-plane API:

import httpx
import os
from modal import Secret
from modal_infra.auth import generate_internal_token

# Assume INTERNAL_CALLBACK_SECRET is set in the Modal environment

internal_secret = os.getenv("INTERNAL_CALLBACK_SECRET")
token = generate_internal_token(internal_secret)

headers = {"Authorization": f"Bearer {token}"}
url = "https://control-plane.example.com/image-builds/trigger/repo/octocat/hello-world"

httpx.post(url, headers=headers).raise_for_status()

Inside the Modal worker, initiating a build sandbox:

# In packages/modal-infra/src/scheduler/image_builder.py

await manager.create_build_sandbox(
    repo_owner=primary.get("repo_owner", ""),
    repo_name=primary.get("repo_name", ""),
    default_branch=primary["branch"],
    clone_token=clone_token,
    user_env_vars=user_env_vars,
    timeout_seconds=sandbox_timeout_seconds,
    repositories=repositories,
)

Scheduler logic for determining rebuild necessity:

# In packages/modal-infra/src/scheduler/image_builder.py

if not matching_ready:
    # No ready image → trigger rebuild

    return True

# Check runtime floor

if min_runtime_version is not None and (version is None or version < min_runtime_version):
    return True

# Check remote SHA drift

remote_sha = _git_ls_remote_sha(
    repo_owner, 
    repo_name,
    f"refs/heads/{base_branch}", 
    clone_token
)
if recorded_by_repo.get((repo_owner.lower(), repo_name.lower())) != remote_sha:
    return True

Summary

• The image prebuild system workflow consists of three phases: Enable & Schedule, Build Execution, and Session Start-up, coordinated between the TypeScript control-plane and Python Modal data-plane.
• Fingerprints deterministically identify image configurations; any structural change invalidates the cache and triggers a new build.
• The scheduler runs every 30 minutes, enforcing a runtime floor and SHA drift checks while capping builds at 8 per tick to prevent resource exhaustion.
• HMAC-authenticated callbacks using generate_internal_token secure communication between Modal workers and the control-plane.
• Session start-up checks packages/control-plane/src/sandbox/lifecycle/image-selection.ts for ready images, falling back to standard provisioning when necessary.

Frequently Asked Questions

What triggers a rebuild in the image prebuild system workflow?

The scheduler initiates rebuilds when three conditions are met: no ready image exists for the current fingerprint, the sandbox runtime version falls below the minimum required floor, or the remote repository’s HEAD SHA differs from the recorded SHA indicating new commits. These checks occur every 30 minutes in packages/modal-infra/src/scheduler/image_builder.py.

How does the system handle build failures?

Failures are captured via the _callback_with_retry mechanism and reported to the control-plane through an authenticated endpoint. The UI displays a red "Failed" status, and the scheduler automatically retries the build on its next tick. Stale or obsolete images are marked as failed and cleaned up to prevent storage bloat.

What is a fingerprint and why does it matter?

A fingerprint is a deterministic hash generated from the ordered list of repositories and their base branches. It matters because it serves as the unique identifier for cached images; any modification to the repository set, order, or branch creates a new fingerprint, ensuring sessions always launch with the correct environment configuration and automatically retiring outdated images.

How is the callback from Modal to the control-plane secured?

Security is enforced through HMAC-signed JWTs created by the generate_internal_token function. The Modal worker signs the payload using the INTERNAL_CALLBACK_SECRET, and the control-plane verifies this signature before accepting build results. This prevents SSRF attacks and ensures that only legitimate workers can report build completion or failure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}