# Running Evaluation Jobs with inspect-ai on Hugging Face Using UV Scripts

> Easily run inspect-ai evaluation jobs on Hugging Face. Our UV scripts support vLLM and Transformers backends, making evaluations simple and secure.

- Repository: [Hugging Face/skills](https://github.com/huggingface/skills)
- Tags: how-to-guide
- Published: 2026-03-08

---

**The huggingface/skills repository provides lightweight UV scripts that wrap inspect-ai evaluations for seamless execution on Hugging Face Jobs, supporting both vLLM and Hugging Face Transformers backends with secure token injection.**

The huggingface/skills repository ships a collection of command-line utilities designed to streamline running inspect-ai evaluations on Hugging Face Jobs using the UV Python runner. These modular scripts handle backend configuration, environment setup, and job submission without requiring persistent infrastructure or complex Docker configurations.

## UV Script Architecture and Core Components

The evaluation suite resides in `skills/hugging-face-evaluation/scripts/` and consists of small, single-purpose Python files (approximately 150 lines each) that can be copied directly into job containers.

### inspect_vllm_uv.py: Backend-Agnostic Evaluation Wrapper

The [`inspect_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/inspect_vllm_uv.py) script serves as the primary entry point for running inspect-ai tasks with configurable backends. Located at [`skills/hugging-face-evaluation/scripts/inspect_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-evaluation/scripts/inspect_vllm_uv.py), this wrapper parses CLI arguments including `--model`, `--task`, and `--backend`, then constructs the appropriate `inspect eval` command.

The script implements `setup_environment()` (lines 41-47) to inject the `HF_TOKEN` secret into the environment as both `HUGGING_FACE_HUB_TOKEN` and `HF_HUB_TOKEN`. For vLLM deployments, it automatically appends backend-specific flags such as `--tensor-parallel-size` and `--gpu-memory-utilization` to the command construction logic (lines 78-106).

### inspect_eval_uv.py: HF Inference Provider Wrapper

For evaluations using the standard Hugging Face Transformers backend, [`skills/hugging-face-evaluation/scripts/inspect_eval_uv.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-evaluation/scripts/inspect_eval_uv.py) provides a simplified wrapper. This script normalizes task names (supporting `suite|task|shots` syntax) and sets the required HF token environment variables (lines 60-66).

Unlike the vLLM variant, this wrapper calls `inspect eval` with fixed parameters `--max-connections 1` and a minimal temperature of `0.001` to prevent out-of-memory errors on job workers (lines 71-86).

### Job Launch Helpers and Orchestration

The repository includes [`run_vllm_eval_job.py`](https://github.com/huggingface/skills/blob/main/run_vllm_eval_job.py) and [`run_eval_job.py`](https://github.com/huggingface/skills/blob/main/run_eval_job.py) as convenience scripts for CI pipelines. These helpers define a **JobSpec** (specifying flavor, GPU type, and secret token) and invoke `hf jobs uv run` to forward arguments to the wrapper scripts. The higher-level [`evaluation_manager.py`](https://github.com/huggingface/skills/blob/main/evaluation_manager.py) provides sub-commands (`inspect-tables`, `extract`, `upload`) that operate on inspect-ai output, enabling automated metric extraction and hub uploads (lines 720-780).

## Executing Evaluations Locally and Remotely

### Local Sanity Checks

Before submitting to remote GPUs, validate your configuration locally using the wrapper scripts directly:

```bash
python inspect_vllm_uv.py \
    --model meta-llama/Llama-3.2-1B \
    --task mmlu \
    --backend vllm \
    --limit 10

```

This executes the evaluation using the specified backend, limits processing to 10 samples, and prints the exact `inspect eval` command being executed for debugging purposes.

### Submitting GPU Jobs via hf jobs uv run

To run evaluations on Hugging Face Jobs infrastructure, use the `hf jobs uv run` command with the `--` separator to distinguish job-level flags from script arguments:

```bash
hf jobs uv run inspect_vllm_uv.py \
    --flavor a10g-small \
    --secret HF_TOKEN=$HF_TOKEN \
    -- \
    --model meta-llama/Llama-3.2-70B \
    --task mmlu \
    --backend vllm \
    --tensor-parallel-size 4 \
    --limit 100

```

The job container automatically installs dependencies listed in [`skills/hugging-face-evaluation/requirements.txt`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-evaluation/requirements.txt) via `uv pip install -r requirements.txt`, ensuring consistent environments across runs.

### Using the Hugging Face Transformers Backend

For models incompatible with vLLM or when avoiding tensor parallelism overhead, use the HF-provider wrapper:

```bash
hf jobs uv run inspect_eval_uv.py \
    --flavor a10g-small \
    --secret HF_TOKEN=$HF_TOKEN \
    -- \
    --model meta-llama/Llama-3.2-1B \
    --task gsm8k \
    --limit 20

```

This approach uses the Hugging Face Transformers inference provider, which offers broader model compatibility than the vLLM backend.

## Secure Token Management

All scripts implement secure token handling without storing secrets in the repository. The `setup_environment()` function extracts the `HF_TOKEN` from job secrets and propagates it to both `HUGGING_FACE_HUB_TOKEN` and `HF_HUB_TOKEN` environment variables. This injection occurs at runtime, ensuring credentials never appear in source code or logs.

## Processing Results with evaluation_manager.py

After job completion, the [`evaluation_manager.py`](https://github.com/huggingface/skills/blob/main/evaluation_manager.py) script enables full pipeline orchestration:

```bash

# Extract metrics from inspect-ai tables

python evaluation_manager.py extract \
    --repo-id my-org/my-model \
    --task mmlu \
    --output metrics.json

# Upload results to the model hub

python evaluation_manager.py upload \
    --repo-id my-org/my-model \
    --metrics-file metrics.json

```

The manager reads inspect-ai table files, normalizes column names, and pushes JSON summaries to the specified model repository for transparent benchmarking.

## Summary

- **Modular architecture**: Each script in `skills/hugging-face-evaluation/scripts/` is intentionally small (≈150 LOC) to facilitate direct deployment into job containers without repository dependencies.
- **Backend flexibility**: Choose between `vllm` for high-throughput tensor-parallel inference or `hf` for broader model compatibility using the same CLI patterns.
- **Secure execution**: The `HF_TOKEN` secret is injected at runtime via `setup_environment()` and mapped to standard Hugging Face environment variables.
- **End-to-end automation**: [`evaluation_manager.py`](https://github.com/huggingface/skills/blob/main/evaluation_manager.py) provides sub-commands for table inspection, metric extraction, and hub uploads, creating reproducible evaluation pipelines.

## Frequently Asked Questions

### How do I select between the vLLM and HF backends when running inspect-ai evaluations?

**Use [`inspect_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/inspect_vllm_uv.py) with `--backend vllm` for high-performance tensor-parallel inference on supported models, or `--backend hf` for standard Hugging Face Transformers execution.** The vLLM backend supports flags like `--tensor-parallel-size` and `--gpu-memory-utilization` for optimizing GPU memory usage, while the HF backend offers broader model compatibility and runs with conservative memory settings (`--max-connections 1` and temperature `0.001`) to prevent OOM errors on job workers.

### What is the purpose of the UV runner in Hugging Face Jobs?

**The UV runner provides a zero-overhead Python execution environment that automatically installs dependencies from [`requirements.txt`](https://github.com/huggingface/skills/blob/main/requirements.txt) without building Docker images.** When you invoke `hf jobs uv run`, the system caches the UV scripts and their dependencies, then executes the evaluation on the specified GPU flavor (such as `a10g-small`), handling environment isolation and resource allocation transparently.

### How does the evaluation_manager.py script process inspect-ai results?

**The script provides three primary sub-commands: `inspect-tables` for viewing results, `extract` for parsing metrics into JSON, and `upload` for publishing to the Hugging Face Hub.** According to the source code (lines 720-780), it reads the table files generated by inspect-ai jobs, normalizes column naming conventions, and can automatically post structured evaluation summaries back to the model repository for version tracking and comparison.

### Where are the runtime dependencies declared for these evaluation scripts?

**All dependencies are listed in [`skills/hugging-face-evaluation/requirements.txt`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-evaluation/requirements.txt), which specifies exact versions of `inspect-ai`, `inspect-evals`, `vllm`, and `torch`.** The UV runner automatically installs these packages when executing `hf jobs uv run`, ensuring that remote job containers match the local development environment without manual image building or package management.