Running Custom Model Evaluations with vLLM and LightEval on Hugging Face Jobs

You can run custom model evaluations using vLLM and LightEval on Hugging Face Jobs by executing PEP 723 scripts that auto-install dependencies and handle GPU provisioning through the hf jobs uv run command.

The huggingface/skills repository provides a complete evaluation workflow through the hugging-face-evaluation skill, enabling high-throughput benchmarking of custom models on GPU hardware without manual environment setup. This workflow leverages PEP 723 script files to automatically manage dependencies when running custom model evaluations with vLLM and LightEval on Hugging Face Jobs.

Core Evaluation Scripts

The skill ships two primary PEP 723 scripts located in skills/hugging-face-evaluation/scripts/:

lighteval_vllm_uv.py

The lighteval_vllm_uv.py script wraps LightEval benchmark suites using the high-throughput vLLM inference engine. It supports both vLLM and Accelerate backends through command-line flags, constructing commands like lighteval vllm <model> <tasks> or lighteval accelerate <model> <tasks> based on the --backend parameter.

inspect_vllm_uv.py

The inspect_vllm_uv.py script executes the inspect-ai benchmark suite, supporting both vLLM and standard Hugging Face Transformers backends. The --backend flag allows seamless switching between vllm/<model> and hf/<model> inference engines, enabling evaluations for models not yet supported by vLLM.

How the Evaluation Workflow Works

Environment Bootstrap and Authentication

Both scripts implement a setup_environment() function that copies the provided HF_TOKEN into HUGGING_FACE_HUB_TOKEN and HF_HUB_TOKEN environment variables. This ensures downstream CLI tools like vLLM, LightEval, and inspect-ai authenticate automatically with the Hugging Face Hub without manual configuration files.

Backend Command Construction

The scripts dynamically construct CLI commands based on parsed arguments:

• LightEval: Assembles lighteval vllm <model> <tasks> or lighteval accelerate <model> <tasks> commands, appending flags like --trust-remote-code, --use-chat-template, and --system-prompt when specified.
• Inspect-ai: Builds inspect eval <task> --model vllm/<model> or inspect eval <task> --model hf/<model> commands, adding parameters such as --temperature, --max-connections, and --tensor-parallel-size.

Execution on Hugging Face Jobs

Because the scripts are pure Python entry points, they integrate seamlessly with the hf jobs uv run wrapper. This wrapper automatically creates an isolated environment, installs the declared dependencies from the PEP 723 header, and provisions the requested hardware flavor (CPU, T4, A10G, A100, etc.).

Running Evaluations Locally and on HF Jobs

Local LightEval with vLLM

Run benchmarks on a local GPU using uv run to handle dependencies automatically:

uv run scripts/lighteval_vllm_uv.py \
    --model meta-llama/Llama-3.2-1B \
    --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" \
    --batch-size 2 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9 \
    --dtype float16 \
    --use-chat-template

This executes MMLU and GSM8K with 5-shot prompting using 16-bit weights and applies the model's chat template.

Submitting to Hugging Face Jobs

Submit the same evaluation to an A10G GPU via Hugging Face Jobs:

HF_TOKEN=$HF_TOKEN \
hf jobs uv run lighteval_vllm_uv.py \
    --flavor a10g-small \
    --secret HF_TOKEN=$HF_TOKEN \
    -- \
    --model "meta-llama/Llama-3.2-1B" \
    --tasks "leaderboard|mmlu|5"

The --flavor flag provisions the hardware, while --secret injects the authentication token securely.

Inspect-ai Benchmarks

Run inspect-ai evaluations locally with vLLM and tensor parallelism:

uv run scripts/inspect_vllm_uv.py \
    --model meta-llama/Llama-3.2-1B \
    --task mmlu \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.85 \
    --trust-remote-code

This distributes the model across two GPUs and enables remote code execution for custom model architectures.

Inspect-ai with Hugging Face Transformers Backend

For models not supported by vLLM, use the HF transformers backend on smaller hardware:

HF_TOKEN=$HF_TOKEN \
hf jobs uv run inspect_vllm_uv.py \
    --flavor t4-small \
    --secret HF_TOKEN=$HF_TOKEN \
    -- \
    --model "google/flan-t5-base" \
    --task gsm8k \
    --backend hf \
    --device cuda \
    --dtype bfloat16

The --backend hf flag switches to the standard transformers inference pipeline.

Hardware Selection and Optimization

The skill includes hardware recommendations based on model parameter count:

Model size	Suggested HF Jobs flavor
< 3B params	t4-small
3 – 13B params	a10g-small
13 – 34B params	a10g-large
≥ 34B params	a100-large

If the selected hardware lacks sufficient VRAM, vLLM raises an OOM error. Mitigate this by adjusting --gpu-memory-utilization (default 0.8) or increasing --tensor-parallel-size to distribute the model across multiple GPUs.

Security and Authentication

The setup_environment() function in both lighteval_vllm_uv.py and inspect_vllm_uv.py handles authentication by propagating HF_TOKEN to HUGGING_FACE_HUB_TOKEN and HF_HUB_TOKEN. Pass the token via environment variable or the --secret flag in HF Jobs; the scripts never log sensitive credentials.

For models requiring custom architecture code (e.g., Phi-2, Qwen), enable --trust-remote-code to allow remote code execution. This flag defaults to False for security and should only be enabled for trusted model repositories.

Summary

• The hugging-face-evaluation skill in huggingface/skills provides PEP 723 scripts for running vLLM and LightEval evaluations on Hugging Face Jobs.
• lighteval_vllm_uv.py and inspect_vllm_uv.py handle environment setup via setup_environment(), construct backend-specific CLI commands, and execute benchmarks via subprocess.run().
• The hf jobs uv run wrapper provisions hardware (T4, A10G, A100), installs dependencies automatically, and injects authentication tokens securely.
• Hardware selection scales from t4-small for models under 3B parameters to a100-large for 34B+ models, with --tensor-parallel-size enabling multi-GPU distribution.

Frequently Asked Questions

What is the difference between LightEval and inspect-ai backends?

LightEval focuses on standard academic benchmarks (MMLU, GSM8K, HellaSwag) using a task format of suite|task|num_fewshot, while inspect-ai provides a framework for agent-based and chat-oriented evaluations with built-in support for multi-turn conversations and tool use. The lighteval_vllm_uv.py script targets LightEval suites, whereas inspect_vllm_uv.py runs inspect-ai tasks.

How do I handle models that require custom code execution?

Pass the --trust-remote-code flag when running either script. This allows vLLM or the HF transformers backend to execute custom architecture code downloaded from the Hugging Face Hub (required for models like Phi-2 or Qwen). For security, this flag defaults to False and should only be enabled for trusted model repositories.

Can I run evaluations on CPU-only hardware?

While the scripts primarily target GPU inference via vLLM, you can run evaluations on CPU-only HF Jobs flavors by using the Accelerate backend in lighteval_vllm_uv.py (pass --backend accelerate) or the HF transformers backend in inspect_vllm_uv.py (pass --backend hf --device cpu). Note that CPU inference is significantly slower and generally suitable only for small models (< 1B parameters) or debugging.

How does the PEP 723 script format benefit HF Jobs execution?

PEP 723 inline metadata allows the scripts to declare Python version requirements (≥ 3.10) and dependencies (lighteval[accelerate,vllm], torch, transformers, vllm) directly in the file header. When launched via uv run, these dependencies install automatically into an isolated environment, eliminating the need for Docker images or manual requirements.txt management on Hugging Face Jobs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}