# How the hf_jobs Tool in ML Intern Manages Cloud Compute and Training Scripts

> Discover how the hf_jobs tool in ML Intern manages cloud compute and training scripts. It translates JSON requests into API calls for efficient execution and real-time log streaming.

- Repository: [Hugging Face/ml-intern](https://github.com/huggingface/ml-intern)
- Tags: how-to-guide
- Published: 2026-04-24

---

**The hf_jobs tool translates JSON requests into Hugging Face API calls, supporting both Python-mode (with UV dependency resolution) and Docker-mode execution, while streaming logs back to the user in real-time.**

The `hf_jobs` tool serves as the bridge between the ML Intern agent and Hugging Face’s cloud-compute platform. Located in [`agent/tools/jobs_tool.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/jobs_tool.py), it encapsulates the full lifecycle of job management—from script resolution and dependency injection to live log streaming and result formatting. This guide explains how the tool orchestrates cloud training workloads using the `huggingface_hub` **HfApi**.

## Core Architecture

### HfJobsTool Class and Operation Dispatch

The **`HfJobsTool`** class (lines 95-106) serves as the central entry point. Upon initialization, it instantiates `HfApi` with the user’s authentication token and stores a reference to the session object. The `execute()` method (lines 120-165) parses the `operation` field from incoming JSON requests and dispatches to the appropriate private handler, ensuring uniform error responses for unknown operations.

### Python-Mode vs Docker-Mode Execution

The tool supports two distinct execution strategies:

**Python-mode** resolves training scripts via `_resolve_uv_command` (lines 221-234), which handles URL references, file paths, or inline code. The method automatically injects `hf-transfer` into dependencies via `_ensure_hf_transfer_dependency` (lines 194-212), constructs a **UV** command (e.g., `uv run /app/train.py`), and encodes inline scripts in base-64 to avoid temporary file creation.

**Docker-mode** bypasses Python-specific preparation, forwarding the user-provided `command` and optional `image` directly to the Hugging Face API (lines 276-285).

### Environment Configuration and Secret Handling

The tool applies a curated set of default environment variables defined in `_DEFAULT_ENV` (lines 27-33) that suppress progress bars and enable high-speed transfers. The `_add_environment_variables` method (lines 44-60) merges user-provided secrets with automatically injected tokens (`HF_TOKEN` / `HUGGINGFACE_HUB_TOKEN`).

### Live Log Streaming and Session Management

Real-time log consumption happens in `_wait_for_job_completion` (lines 82-124), which spawns a background thread to pull from the synchronous `fetch_job_logs` generator. This thread pushes log lines onto an `asyncio.Queue`, while the async consumer forwards them to the frontend via `log_callback`. The implementation includes retry logic for network failures (up to 100 attempts) and polls job status to detect terminal states early.

Session-level tracking occurs through `_running_job_ids`, managed in `_run_job` (lines 420-428), enabling graceful cancellation of all active jobs when the user interrupts the session.

## End-to-End Job Execution Flow

When the agent receives a training request, the following sequence executes:

1. **Request Parsing**: The `hf_jobs_handler` (lines 558-585) instantiates `HfJobsTool`, injects the user token, and calls `tool.execute()`.

2. **Script Resolution**: For sandbox-resident files, the handler reads content via `resolve_sandbox_script`. The tool then determines execution mode and potentially wraps the script using `_wrap_inline_script` for base-64 encoding.

3. **Dependency Injection**: The tool guarantees `hf-transfer` availability through `_ensure_hf_transfer_dependency`, adding it to the user’s dependency list if absent.

4. **Job Submission**: `_run_job` calls `self.api.run_job()` (lines 332-350) with the resolved command, container image, environment variables, secrets, hardware flavor (e.g., `a100-large`), and timeout duration.

5. **Log Streaming**: `_wait_for_job_completion` streams logs in real-time, filtering UV installation noise via `_filter_uv_install_output` and stripping ANSI codes before presentation.

6. **Result Formatting**: Raw `JobInfo` objects convert to dictionaries via `_job_info_to_dict`, then format as markdown tables using utilities from [`agent/tools/utilities.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/utilities.py).

## Supported Operations and Usage Examples

### Submitting Training Jobs

**Python-mode execution** requires a script path and dependency list:

```json
{
  "operation": "run",
  "script": "/app/train.py",
  "dependencies": ["transformers", "torch", "datasets"],
  "hardware_flavor": "a100-large",
  "timeout": "8h"
}

```

This triggers UV-based dependency resolution and launches an A100-large instance for up to eight hours.

**Docker-mode execution** accepts arbitrary commands:

```json
{
  "operation": "run",
  "command": "python /app/train.py --epochs 10",
  "image": "pytorch/pytorch:latest",
  "hardware_flavor": "t4-small"
}

```

### Monitoring and Managing Jobs

List active jobs:

```json
{
  "operation": "ps"
}

```

Fetch logs for a specific job:

```json
{
  "operation": "logs",
  "job_id": "abc123def456"
}

```

Cancel a running job:

```json
{
  "operation": "cancel",
  "job_id": "abc123def456"
}

```

### Scheduled Job Workflows

Create recurring training tasks:

```json
{
  "operation": "scheduled run",
  "script": "/app/train.py",
  "dependencies": ["transformers", "torch"],
  "schedule": "@daily",
  "hardware_flavor": "t4-small",
  "timeout": "4h"
}

```

List all scheduled jobs including suspended ones:

```json
{
  "operation": "scheduled ps",
  "all": true
}

```

## Key Implementation Files

- **[`agent/tools/jobs_tool.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/jobs_tool.py)**: Implements the complete `hf_jobs` lifecycle from request parsing through API interaction to result formatting.
- **[`agent/tools/utilities.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/utilities.py)**: Provides markdown formatters like `format_jobs_table` and `format_job_details` for human-readable output.
- **[`agent/tools/__init__.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/__init__.py)**: Registers `HF_JOBS_TOOL_SPEC` (lines 446-556) to expose the JSON schema to the agent framework.

## Summary

- The **hf_jobs** tool in [`agent/tools/jobs_tool.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/jobs_tool.py) acts as a bridge between the ML Intern agent and Hugging Face cloud compute.
- **Python-mode** uses UV for dependency resolution and base-64 encoding for inline scripts, while **Docker-mode** executes raw commands in specified containers.
- **Real-time log streaming** operates via background threads and `asyncio.Queue`, with automatic retries for network resilience.
- The tool automatically injects `hf-transfer` and Hugging Face tokens into the environment.
- **Session tracking** via `_running_job_ids` enables bulk cancellation of active jobs on interrupt.
- All operations return markdown-formatted results using helper functions from [`agent/tools/utilities.py`](https://github.com/huggingface/ml-intern/blob/main/agent/tools/utilities.py).

## Frequently Asked Questions

### What is the difference between Python-mode and Docker-mode in hf_jobs?

**Python-mode** resolves scripts and dependencies locally using UV, automatically adding `hf-transfer` to the dependency list and encoding inline scripts in base-64 to avoid temporary files. **Docker-mode** forwards the user-provided command and image directly to the Hugging Face API without Python-specific preprocessing, suitable for custom container workflows or non-Python executables.

### How does hf_jobs handle Python dependencies?

The tool calls `_ensure_hf_transfer_dependency` to guarantee `hf-transfer` is present in the dependency list, then constructs a UV command via `_resolve_uv_command`. This command executes as `uv run [script]`, allowing UV to resolve and install the exact dependency tree before starting the training script.

### How are logs streamed in real-time from Hugging Face cloud jobs?

The `_wait_for_job_completion` method spawns a background thread that consumes the synchronous `fetch_job_logs` generator from `huggingface_hub`, pushes lines onto an `asyncio.Queue`, and forwards them to the frontend through the `log_callback` parameter. The implementation includes retry logic with up to 100 attempts for network failures and polls job status to detect completion early.

### Can hf_jobs run scheduled training tasks?

Yes, the tool supports scheduled operations through `scheduled run`, `scheduled ps`, and `scheduled cancel` operations. These map to `HfApi.create_scheduled_job()` and related endpoints, allowing users to define cron-like schedules (e.g., `@daily`) for recurring training jobs on specific hardware flavors.