# Running Custom Model Evaluations with vLLM and LightEval on Hugging Face Jobs

> Effortlessly run custom model evaluations with vLLM and LightEval on Hugging Face Jobs. Auto-install dependencies and provision GPUs using the hf jobs uv run command.

- Repository: [Hugging Face/skills](https://github.com/huggingface/skills)
- Tags: tutorial
- Published: 2026-03-08

---

**You can run custom model evaluations using vLLM and LightEval on Hugging Face Jobs by executing PEP 723 scripts that auto-install dependencies and handle GPU provisioning through the `hf jobs uv run` command.**

The `huggingface/skills` repository provides a complete evaluation workflow through the **hugging-face-evaluation** skill, enabling high-throughput benchmarking of custom models on GPU hardware without manual environment setup. This workflow leverages PEP 723 script files to automatically manage dependencies when running custom model evaluations with vLLM and LightEval on Hugging Face Jobs.

## Core Evaluation Scripts

The skill ships two primary PEP 723 scripts located in `skills/hugging-face-evaluation/scripts/`:

### lighteval_vllm_uv.py

The [`lighteval_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/lighteval_vllm_uv.py) script wraps **LightEval** benchmark suites using the high-throughput **vLLM** inference engine. It supports both vLLM and Accelerate backends through command-line flags, constructing commands like `lighteval vllm <model> <tasks>` or `lighteval accelerate <model> <tasks>` based on the `--backend` parameter.

### inspect_vllm_uv.py

The [`inspect_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/inspect_vllm_uv.py) script executes the **inspect-ai** benchmark suite, supporting both vLLM and standard Hugging Face Transformers backends. The `--backend` flag allows seamless switching between `vllm/<model>` and `hf/<model>` inference engines, enabling evaluations for models not yet supported by vLLM.

## How the Evaluation Workflow Works

### Environment Bootstrap and Authentication

Both scripts implement a `setup_environment()` function that copies the provided `HF_TOKEN` into `HUGGING_FACE_HUB_TOKEN` and `HF_HUB_TOKEN` environment variables. This ensures downstream CLI tools like vLLM, LightEval, and inspect-ai authenticate automatically with the Hugging Face Hub without manual configuration files.

### Backend Command Construction

The scripts dynamically construct CLI commands based on parsed arguments:

- **LightEval**: Assembles `lighteval vllm <model> <tasks>` or `lighteval accelerate <model> <tasks>` commands, appending flags like `--trust-remote-code`, `--use-chat-template`, and `--system-prompt` when specified.
- **Inspect-ai**: Builds `inspect eval <task> --model vllm/<model>` or `inspect eval <task> --model hf/<model>` commands, adding parameters such as `--temperature`, `--max-connections`, and `--tensor-parallel-size`.

### Execution on Hugging Face Jobs

Because the scripts are pure Python entry points, they integrate seamlessly with the `hf jobs uv run` wrapper. This wrapper automatically creates an isolated environment, installs the declared dependencies from the PEP 723 header, and provisions the requested hardware flavor (CPU, T4, A10G, A100, etc.).

## Running Evaluations Locally and on HF Jobs

### Local LightEval with vLLM

Run benchmarks on a local GPU using `uv run` to handle dependencies automatically:

```bash
uv run scripts/lighteval_vllm_uv.py \
    --model meta-llama/Llama-3.2-1B \
    --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \
    --batch-size 2 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9 \
    --dtype float16 \
    --use-chat-template

```

This executes MMLU and GSM8K with 5-shot prompting using 16-bit weights and applies the model's chat template.

### Submitting to Hugging Face Jobs

Submit the same evaluation to an A10G GPU via Hugging Face Jobs:

```bash
HF_TOKEN=$HF_TOKEN \
hf jobs uv run lighteval_vllm_uv.py \
    --flavor a10g-small \
    --secret HF_TOKEN=$HF_TOKEN \
    -- \
    --model "meta-llama/Llama-3.2-1B" \
    --tasks "leaderboard|mmlu|5"

```

The `--flavor` flag provisions the hardware, while `--secret` injects the authentication token securely.

### Inspect-ai Benchmarks

Run inspect-ai evaluations locally with vLLM and tensor parallelism:

```bash
uv run scripts/inspect_vllm_uv.py \
    --model meta-llama/Llama-3.2-1B \
    --task mmlu \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.85 \
    --trust-remote-code

```

This distributes the model across two GPUs and enables remote code execution for custom model architectures.

### Inspect-ai with Hugging Face Transformers Backend

For models not supported by vLLM, use the HF transformers backend on smaller hardware:

```bash
HF_TOKEN=$HF_TOKEN \
hf jobs uv run inspect_vllm_uv.py \
    --flavor t4-small \
    --secret HF_TOKEN=$HF_TOKEN \
    -- \
    --model "google/flan-t5-base" \
    --task gsm8k \
    --backend hf \
    --device cuda \
    --dtype bfloat16

```

The `--backend hf` flag switches to the standard transformers inference pipeline.

## Hardware Selection and Optimization

The skill includes hardware recommendations based on model parameter count:

| Model size | Suggested HF Jobs flavor |
|------------|--------------------------|
| < 3B params | `t4-small` |
| 3 – 13B params | `a10g-small` |
| 13 – 34B params | `a10g-large` |
| ≥ 34B params | `a100-large` |

If the selected hardware lacks sufficient VRAM, vLLM raises an OOM error. Mitigate this by adjusting `--gpu-memory-utilization` (default 0.8) or increasing `--tensor-parallel-size` to distribute the model across multiple GPUs.

## Security and Authentication

The `setup_environment()` function in both [`lighteval_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/lighteval_vllm_uv.py) and [`inspect_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/inspect_vllm_uv.py) handles authentication by propagating `HF_TOKEN` to `HUGGING_FACE_HUB_TOKEN` and `HF_HUB_TOKEN`. Pass the token via environment variable or the `--secret` flag in HF Jobs; the scripts never log sensitive credentials.

For models requiring custom architecture code (e.g., Phi-2, Qwen), enable `--trust-remote-code` to allow remote code execution. This flag defaults to `False` for security and should only be enabled for trusted model repositories.

## Summary

- The **hugging-face-evaluation** skill in `huggingface/skills` provides PEP 723 scripts for running **vLLM** and **LightEval** evaluations on Hugging Face Jobs.
- **[`lighteval_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/lighteval_vllm_uv.py)** and **[`inspect_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/inspect_vllm_uv.py)** handle environment setup via `setup_environment()`, construct backend-specific CLI commands, and execute benchmarks via `subprocess.run()`.
- The **`hf jobs uv run`** wrapper provisions hardware (T4, A10G, A100), installs dependencies automatically, and injects authentication tokens securely.
- Hardware selection scales from `t4-small` for models under 3B parameters to `a100-large` for 34B+ models, with `--tensor-parallel-size` enabling multi-GPU distribution.

## Frequently Asked Questions

### What is the difference between LightEval and inspect-ai backends?

**LightEval** focuses on standard academic benchmarks (MMLU, GSM8K, HellaSwag) using a task format of `suite|task|num_fewshot`, while **inspect-ai** provides a framework for agent-based and chat-oriented evaluations with built-in support for multi-turn conversations and tool use. The [`lighteval_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/lighteval_vllm_uv.py) script targets LightEval suites, whereas [`inspect_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/inspect_vllm_uv.py) runs inspect-ai tasks.

### How do I handle models that require custom code execution?

Pass the `--trust-remote-code` flag when running either script. This allows vLLM or the HF transformers backend to execute custom architecture code downloaded from the Hugging Face Hub (required for models like Phi-2 or Qwen). For security, this flag defaults to `False` and should only be enabled for trusted model repositories.

### Can I run evaluations on CPU-only hardware?

While the scripts primarily target GPU inference via vLLM, you can run evaluations on CPU-only HF Jobs flavors by using the **Accelerate** backend in [`lighteval_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/lighteval_vllm_uv.py) (pass `--backend accelerate`) or the **HF transformers** backend in [`inspect_vllm_uv.py`](https://github.com/huggingface/skills/blob/main/inspect_vllm_uv.py) (pass `--backend hf --device cpu`). Note that CPU inference is significantly slower and generally suitable only for small models (< 1B parameters) or debugging.

### How does the PEP 723 script format benefit HF Jobs execution?

PEP 723 inline metadata allows the scripts to declare Python version requirements (≥ 3.10) and dependencies (`lighteval[accelerate,vllm]`, `torch`, `transformers`, `vllm`) directly in the file header. When launched via `uv run`, these dependencies install automatically into an isolated environment, eliminating the need for Docker images or manual [`requirements.txt`](https://github.com/huggingface/skills/blob/main/requirements.txt) management on Hugging Face Jobs.