Running Evaluation Jobs with inspect-ai on Hugging Face Using UV Scripts

The huggingface/skills repository provides lightweight UV scripts that wrap inspect-ai evaluations for seamless execution on Hugging Face Jobs, supporting both vLLM and Hugging Face Transformers backends with secure token injection.

The huggingface/skills repository ships a collection of command-line utilities designed to streamline running inspect-ai evaluations on Hugging Face Jobs using the UV Python runner. These modular scripts handle backend configuration, environment setup, and job submission without requiring persistent infrastructure or complex Docker configurations.

UV Script Architecture and Core Components

The evaluation suite resides in skills/hugging-face-evaluation/scripts/ and consists of small, single-purpose Python files (approximately 150 lines each) that can be copied directly into job containers.

inspect_vllm_uv.py: Backend-Agnostic Evaluation Wrapper

The inspect_vllm_uv.py script serves as the primary entry point for running inspect-ai tasks with configurable backends. Located at skills/hugging-face-evaluation/scripts/inspect_vllm_uv.py, this wrapper parses CLI arguments including --model, --task, and --backend, then constructs the appropriate inspect eval command.

The script implements setup_environment() (lines 41-47) to inject the HF_TOKEN secret into the environment as both HUGGING_FACE_HUB_TOKEN and HF_HUB_TOKEN. For vLLM deployments, it automatically appends backend-specific flags such as --tensor-parallel-size and --gpu-memory-utilization to the command construction logic (lines 78-106).

inspect_eval_uv.py: HF Inference Provider Wrapper

For evaluations using the standard Hugging Face Transformers backend, skills/hugging-face-evaluation/scripts/inspect_eval_uv.py provides a simplified wrapper. This script normalizes task names (supporting suite|task|shots syntax) and sets the required HF token environment variables (lines 60-66).

Unlike the vLLM variant, this wrapper calls inspect eval with fixed parameters --max-connections 1 and a minimal temperature of 0.001 to prevent out-of-memory errors on job workers (lines 71-86).

Job Launch Helpers and Orchestration

The repository includes run_vllm_eval_job.py and run_eval_job.py as convenience scripts for CI pipelines. These helpers define a JobSpec (specifying flavor, GPU type, and secret token) and invoke hf jobs uv run to forward arguments to the wrapper scripts. The higher-level evaluation_manager.py provides sub-commands (inspect-tables, extract, upload) that operate on inspect-ai output, enabling automated metric extraction and hub uploads (lines 720-780).

Executing Evaluations Locally and Remotely

Local Sanity Checks

Before submitting to remote GPUs, validate your configuration locally using the wrapper scripts directly:

python inspect_vllm_uv.py \
    --model meta-llama/Llama-3.2-1B \
    --task mmlu \
    --backend vllm \
    --limit 10

This executes the evaluation using the specified backend, limits processing to 10 samples, and prints the exact inspect eval command being executed for debugging purposes.

Submitting GPU Jobs via hf jobs uv run

To run evaluations on Hugging Face Jobs infrastructure, use the hf jobs uv run command with the -- separator to distinguish job-level flags from script arguments:

hf jobs uv run inspect_vllm_uv.py \
    --flavor a10g-small \
    --secret HF_TOKEN=$HF_TOKEN \
    -- \
    --model meta-llama/Llama-3.2-70B \
    --task mmlu \
    --backend vllm \
    --tensor-parallel-size 4 \
    --limit 100

The job container automatically installs dependencies listed in skills/hugging-face-evaluation/requirements.txt via uv pip install -r requirements.txt, ensuring consistent environments across runs.

Using the Hugging Face Transformers Backend

For models incompatible with vLLM or when avoiding tensor parallelism overhead, use the HF-provider wrapper:

hf jobs uv run inspect_eval_uv.py \
    --flavor a10g-small \
    --secret HF_TOKEN=$HF_TOKEN \
    -- \
    --model meta-llama/Llama-3.2-1B \
    --task gsm8k \
    --limit 20

This approach uses the Hugging Face Transformers inference provider, which offers broader model compatibility than the vLLM backend.

Secure Token Management

All scripts implement secure token handling without storing secrets in the repository. The setup_environment() function extracts the HF_TOKEN from job secrets and propagates it to both HUGGING_FACE_HUB_TOKEN and HF_HUB_TOKEN environment variables. This injection occurs at runtime, ensuring credentials never appear in source code or logs.

Processing Results with evaluation_manager.py

After job completion, the evaluation_manager.py script enables full pipeline orchestration:

# Extract metrics from inspect-ai tables

python evaluation_manager.py extract \
    --repo-id my-org/my-model \
    --task mmlu \
    --output metrics.json

# Upload results to the model hub

python evaluation_manager.py upload \
    --repo-id my-org/my-model \
    --metrics-file metrics.json

The manager reads inspect-ai table files, normalizes column names, and pushes JSON summaries to the specified model repository for transparent benchmarking.

Summary

• Modular architecture: Each script in skills/hugging-face-evaluation/scripts/ is intentionally small (≈150 LOC) to facilitate direct deployment into job containers without repository dependencies.
• Backend flexibility: Choose between vllm for high-throughput tensor-parallel inference or hf for broader model compatibility using the same CLI patterns.
• Secure execution: The HF_TOKEN secret is injected at runtime via setup_environment() and mapped to standard Hugging Face environment variables.
• End-to-end automation: evaluation_manager.py provides sub-commands for table inspection, metric extraction, and hub uploads, creating reproducible evaluation pipelines.

Frequently Asked Questions

How do I select between the vLLM and HF backends when running inspect-ai evaluations?

Use inspect_vllm_uv.py with --backend vllm for high-performance tensor-parallel inference on supported models, or --backend hf for standard Hugging Face Transformers execution. The vLLM backend supports flags like --tensor-parallel-size and --gpu-memory-utilization for optimizing GPU memory usage, while the HF backend offers broader model compatibility and runs with conservative memory settings (--max-connections 1 and temperature 0.001) to prevent OOM errors on job workers.

What is the purpose of the UV runner in Hugging Face Jobs?

The UV runner provides a zero-overhead Python execution environment that automatically installs dependencies from requirements.txt without building Docker images. When you invoke hf jobs uv run, the system caches the UV scripts and their dependencies, then executes the evaluation on the specified GPU flavor (such as a10g-small), handling environment isolation and resource allocation transparently.

How does the evaluation_manager.py script process inspect-ai results?

The script provides three primary sub-commands: inspect-tables for viewing results, extract for parsing metrics into JSON, and upload for publishing to the Hugging Face Hub. According to the source code (lines 720-780), it reads the table files generated by inspect-ai jobs, normalizes column naming conventions, and can automatically post structured evaluation summaries back to the model repository for version tracking and comparison.

Where are the runtime dependencies declared for these evaluation scripts?

All dependencies are listed in skills/hugging-face-evaluation/requirements.txt, which specifies exact versions of inspect-ai, inspect-evals, vllm, and torch. The UV runner automatically installs these packages when executing hf jobs uv run, ensuring that remote job containers match the local development environment without manual image building or package management.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}