Handling Authentication for Pushing Models and Datasets to Hugging Face Hub

Authenticate with the Hugging Face Hub by retrieving your token from environment variables or cached credentials, then call huggingface_hub.login() to establish a persistent session before pushing models or datasets.

The huggingface/skills repository centralizes production-grade patterns for handling authentication when pushing models and datasets to the Hugging Face Hub. These implementations demonstrate secure token management across local development, CI pipelines, and cloud job environments without hardcoding credentials.

Three Authentication Mechanisms

The Skills repository implements three complementary approaches to Hugging Face Hub authentication, each suited to different execution contexts.

1. Explicit Programmatic Login with huggingface_hub.login

The most direct method uses huggingface_hub.login(token=...) to perform the same OAuth handshake as the CLI, caching credentials locally for subsequent API calls. In skills/hugging-face-model-trainer/scripts/unsloth_sft_example.py (lines 62‑70), the script reads HF_TOKEN or hfjob from the environment and establishes a session:

from huggingface_hub import login
import os

token = os.getenv("HF_TOKEN") or os.getenv("hfjob")
login(token=token)

This creates the ~/.huggingface/token file that the Hub SDK automatically detects for all future operations.

2. Automatic Token Discovery via get_token()

For environments where tokens may be pre-authenticated, huggingface_hub.get_token() provides robust fallback logic. The script at skills/hugging-face-jobs/scripts/generate-responses.py (lines 27‑39) demonstrates this pattern by checking CLI arguments, environment variables, and finally cached logins:

from huggingface_hub import get_token, login

HF_TOKEN = args.hf_token or os.getenv("HF_TOKEN") or get_token()
login(token=HF_TOKEN)

This method reads from stored logins (e.g., from huggingface-cli login) or the ~/.huggingface/token file, enabling seamless execution in CI runners and notebooks.

3. Direct Token Passing to Low-Level APIs

When working with private datasets or specialized integrations like DuckDB, tokens must be injected at the API level. The HFDatasetSQL class in skills/hugging-face-datasets/scripts/sql_manager.py (lines 54‑95) stores self.token initialized from HF_TOKEN and creates a DuckDB secret for private dataset access:

CREATE SECRET hf_token (TYPE HUGGINGFACE, TOKEN '...');

When calling push_to_hub, the class explicitly forwards the token (lines 71‑81):

dataset.push_to_hub(..., token=self.token)

This guarantees authentication even when the global session cache is unavailable.

Core Authentication Flow

According to the source code in huggingface/skills, the authentication logic follows a strict priority hierarchy:

1. Environment variable priority – All entry points first check for HF_TOKEN (or the legacy hfjob variable used by the Jobs platform).
2. Fallback to stored login – If no environment variable exists, huggingface_hub.get_token() reads the cached token from ~/.huggingface/token.
3. Explicit login() invocation – The resolved token is passed to login(token=...), creating a local credential file for automatic SDK detection.
4. Token propagation – High-level SDK calls like Dataset.push_to_hub() inherit the cached credential, while specialized managers (like the SQL handler) inject tokens directly into DuckDB secrets or API calls.

This layered approach ensures robust authentication across local development, cloud jobs, and containerized evaluations.

Implementation Examples

Programmatic Login Before Model Upload

Use this pattern when training and pushing models in scripts like unsloth_sft_example.py:

import os
from huggingface_hub import login
from transformers import AutoModelForCausalLM, AutoTokenizer

# Resolve token from environment

token = os.getenv("HF_TOKEN") or os.getenv("hfjob")
if not token:
    raise RuntimeError("Set HF_TOKEN before pushing a model")

# Establish cached session

login(token=token)

# Push model and tokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model.push_to_hub("username/my-gpt2", token=token)
tokenizer.push_to_hub("username/my-gpt2", token=token)

Automatic Token Discovery in Job Scripts

For reusable job infrastructure as implemented in generate-responses.py:

import argparse
import os
import sys
from huggingface_hub import get_token, login

parser = argparse.ArgumentParser()
parser.add_argument("--hf-token", help="HF token (optional)")
args = parser.parse_args()

# Resolve token: CLI arg → env var → cached login

HF_TOKEN = args.hf_token or os.getenv("HF_TOKEN") or get_token()
if not HF_TOKEN:
    sys.exit("No HF token found – aborting")

login(token=HF_TOKEN)
print("Authenticated with Hub")

Pushing Datasets with the SQL Manager

For SQL-driven workflows using the HFDatasetSQL class from sql_manager.py:

from sql_manager import HFDatasetSQL

# Initialize with automatic HF_TOKEN detection

sql = HFDatasetSQL()

# Transform data via SQL

query = "SELECT subject, COUNT(*) AS cnt FROM data GROUP BY subject"
results = sql.query("cais/mmlu", query, split="train", output_format="df")

# Push to Hub with explicit authentication

sql.push_to_hub(
    dataset_id="cais/mmlu",
    target_repo="my-org/nutrition-subset",
    sql=query,
    private=False,
    commit_message="Create nutrition subset",
)

Key Files and Authentication Patterns

Path	Role	Implementation Detail
skills/hugging-face-model-trainer/scripts/unsloth_sft_example.py	Model training authentication	Lines 62‑70: Env-var token retrieval + login()
skills/hugging-face-jobs/scripts/generate-responses.py	Job infrastructure	Lines 27‑39: get_token() fallback logic
skills/hugging-face-datasets/scripts/sql_manager.py	Dataset SQL integration	Lines 54‑95: DuckDB secret creation; lines 71‑81: push_to_hub with token
skills/hugging-face-paper-publisher/scripts/paper_manager.py	Alternative token source	Uses HfFolder.get_token() for cached credentials
skills/hugging-face-evaluation/scripts/run_eval_job.py	Container job security	Propagates --secrets HF_TOKEN=... to evaluation containers

Summary

• Store tokens securely using the HF_TOKEN environment variable rather than hardcoding values in scripts.
• Use huggingface_hub.login() to establish a persistent session that caches credentials at ~/.huggingface/token for subsequent API calls.
• Implement fallback chains with get_token() to support both interactive development (cached logins) and automated environments (CI/CD).
• Pass tokens explicitly when working with low-level integrations like DuckDB or private dataset operations to ensure authentication bypasses the global cache.
• Reference the huggingface/skills repository for production patterns that handle edge cases in cloud job platforms and containerized training.

Frequently Asked Questions

How do I authenticate without running huggingface-cli login?

Set the HF_TOKEN environment variable in your shell or CI configuration, then call huggingface_hub.login(token=os.getenv("HF_TOKEN")) at the start of your script. This programmatic approach mirrors the CLI authentication flow without requiring interactive terminal input.

Why does my script fail to push private datasets even after logging in?

Some low-level integrations, such as the DuckDB-based SQL manager in sql_manager.py, require explicit token passing rather than relying on the global cache. Ensure you forward the token directly to Dataset.push_to_hub(token=...) or create the appropriate DuckDB secret using CREATE SECRET hf_token.

What is the difference between get_token() and HfFolder.get_token()?

Both methods read the cached token from ~/.huggingface/token, but huggingface_hub.get_token() is the modern, recommended API that handles additional edge cases and environment variable checks. The HfFolder class (used in paper_manager.py) provides legacy access to the same storage mechanism.

How do I handle authentication in containerized evaluation jobs?

Pass the token via container secrets as shown in run_eval_job.py and run_vllm_eval_job.py. Use the --secrets HF_TOKEN=... pattern to inject the credential into the job environment, then retrieve it inside the container using os.getenv("HF_TOKEN") before calling login().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}