# Creating and Managing Hugging Face Hub Datasets with System Prompts

> Easily create and manage Hugging Face Hub datasets with system prompts. Configure and populate datasets directly in repository config files using JSON templates.

- Repository: [Hugging Face/skills](https://github.com/huggingface/skills)
- Tags: how-to-guide
- Published: 2026-03-08

---

**The hugging-face-datasets skill in the huggingface/skills repository enables you to create, configure, and populate Hugging Face Hub datasets while embedding system prompts directly into repository configuration files using JSON templates.**

Managing machine learning datasets requires consistent schema enforcement and embedded instructions for data generation or model fine-tuning. The huggingface/skills repository provides a dedicated **hugging-face-datasets** skill that streamlines creating and managing Hugging Face Hub datasets with system prompts, allowing you to initialize repositories, validate row structures against templates, and persist system-level instructions in version-controlled [`config.json`](https://github.com/huggingface/skills/blob/main/config.json) files.

## Architecture of the Hugging Face Datasets Skill

The skill is organized into three logical layers that handle command parsing, Hub API interactions, and template management.

### CLI Entry Point and Command Routing

The command-line interface is implemented in [`skills/hugging-face-datasets/scripts/dataset_manager.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-datasets/scripts/dataset_manager.py). The `if __name__ == "__main__"` block (lines 25‑50) constructs an `argparse` hierarchy that exposes subcommands including `init`, `quick_setup`, `add_rows`, and `stats`. When you invoke a command, the parser forwards arguments to the appropriate library function, such as `quick_setup()` for automated repository initialization.

### Dataset Manager Core Functions

The core logic for Hub interactions resides in functions like `init_dataset()`, `define_config()`, and `add_rows()` within the same file. These wrappers around the `huggingface_hub` library handle repository creation, metadata upload, and JSON‑Lines streaming. For example, `define_config()` (lines 74‑95) writes the [`config.json`](https://github.com/huggingface/skills/blob/main/config.json) file that stores dataset metadata and system prompts, while `add_rows()` (lines 55‑63) validates incoming data against the template schema before appending to the dataset.

### Template System for System Prompts

JSON templates defining common data shapes—such as chat, classification, and QA—are stored in `skills/hugging-face-datasets/templates/*.json`. Each template includes a `system_prompt` field that describes the intended AI behavior for data generated under that schema. When you select a template via the `--template` argument, the skill automatically extracts this prompt and persists it in the repository configuration.

## How System Prompts Are Embedded

System prompts are embedded into datasets through a structured workflow that bridges template selection and configuration persistence.

### Selecting and Loading Templates

When you run `quick_setup` with the `--template` flag, the skill invokes `load_dataset_template()` (line 97) to read the corresponding JSON file from the `templates` directory. For instance, selecting `--template chat` loads [`templates/chat.json`](https://github.com/huggingface/skills/blob/main/templates/chat.json), which contains a predefined schema and a `system_prompt` value such as `"You are an AI assistant …"`.

### Persisting Prompts in config.json

After loading the template, `quick_setup()` extracts the system prompt using `template_config.get("system_prompt", "")` (line 38) and passes it to `define_config()`. This function writes a [`config.json`](https://github.com/huggingface/skills/blob/main/config.json) file (lines 84‑90) that includes a `"system_prompt"` key alongside other metadata. Downstream tools—such as fine‑tuning scripts or evaluation pipelines—can read this file to ensure generated data adheres to the described behavior.

## Practical Workflow: Creating and Managing Datasets

The skill supports a complete lifecycle for dataset management, from repository creation to data ingestion and transformation.

### Initialize a Dataset with quick_setup

The fastest way to create a new dataset with an embedded system prompt is using the `quick_setup` subcommand:

```bash
uv run skills/hugging-face-datasets/scripts/dataset_manager.py quick_setup \
    --repo_id "myuser/my-chat-dataset" \
    --template chat

```

This command creates the repository `myuser/my-chat-dataset` on the Hugging Face Hub, uploads a `README` and [`config.json`](https://github.com/huggingface/skills/blob/main/config.json) containing the chat system prompt from [`templates/chat.json`](https://github.com/huggingface/skills/blob/main/templates/chat.json), and optionally seeds the dataset with example rows if the template includes them.

### Validate and Stream Rows

To append new data while enforcing the template schema, use the `add_rows` subcommand:

```bash

# Prepare a JSON‑Lines file with new examples

cat <<EOF > new_rows.jsonl
{"messages":[{"role":"user","content":"How do I bake a cake?"},{"role":"assistant","content":"Here is a recipe …"}]}
EOF

# Stream validated rows into the dataset

uv run skills/hugging-face-datasets/scripts/dataset_manager.py add_rows \
    --repo_id "myuser/my-chat-dataset" \
    --split train \
    --template chat \
    --rows_json "$(cat new_rows.jsonl)"

```

The `add_rows` function calls `validate_by_template()` (lines 110‑150) to ensure every row conforms to the chat schema before uploading. If validation fails, the script aborts and reports the specific error without uploading partial data to the Hub.

### Query and Transform with SQL Manager

For advanced workflows, the [`sql_manager.py`](https://github.com/huggingface/skills/blob/main/sql_manager.py) script enables DuckDB‑powered querying and transformation of existing datasets:

```bash
uv run skills/hugging-face-datasets/scripts/sql_manager.py query \
    --dataset "cais/mmlu" \
    --sql "SELECT * FROM data WHERE subject='nutrition'" \
    --push-to "myuser/nutrition-subset"

```

This command builds an `hf://` protocol path, executes the SQL query, and invokes `push_to_hub()` (lines 332‑365) to write the filtered results to a new repository. The SQL manager reuses the same Hugging Face token and Hub API patterns as the dataset manager, ensuring consistent authentication across the skill suite.

## Programmatic Python API

You can also embed dataset creation and system prompt management directly into Python training pipelines:

```python
from skills.hugging-face-datasets.scripts.dataset_manager import define_config
import json
import pathlib

# Define target repository

repo_id = "myuser/my-chat-dataset"

# Load system prompt from template

template_path = pathlib.Path(__file__).parent.parent / "templates" / "chat.json"
prompt = json.loads(template_path.read_text())["system_prompt"]

# Persist configuration with embedded system prompt

define_config(repo_id, system_prompt=prompt)

```

This approach allows fine‑tuning frameworks to dynamically read the `system_prompt` from [`config.json`](https://github.com/huggingface/skills/blob/main/config.json) and generate training examples that adhere to the specified behavior.

## Summary

- The **hugging-face-datasets** skill provides a CLI and Python API for creating and managing Hugging Face Hub datasets with system prompts.
- **System prompts** are embedded via JSON templates (`templates/*.json`) and persisted in repository [`config.json`](https://github.com/huggingface/skills/blob/main/config.json) files through `define_config()`.
- The **[`dataset_manager.py`](https://github.com/huggingface/skills/blob/main/dataset_manager.py)** script handles repository initialization (`quick_setup`), row validation (`validate_by_template` at lines 110‑150), and data streaming (`add_rows`).
- The **[`sql_manager.py`](https://github.com/huggingface/skills/blob/main/sql_manager.py)** script enables DuckDB‑based querying and subset creation via `push_to_hub()` (lines 332‑365).
- All workflows leverage the `huggingface_hub` API and reuse authentication tokens across the skill suite.

## Frequently Asked Questions

### How do I embed a custom system prompt when creating a new dataset?

You can embed a custom system prompt by using the `quick_setup` command with a template that contains your desired prompt, or by manually calling `define_config()` in Python. The skill stores the prompt in [`config.json`](https://github.com/huggingface/skills/blob/main/config.json) at the repository root, making it accessible to downstream training scripts. If you need a completely custom prompt, create a new JSON file in the `templates/` directory with a `"system_prompt"` key and pass it to `--template`.

### What validation does the skill perform when adding rows to a dataset?

When you use the `add_rows` subcommand or call the function directly, the skill invokes `validate_by_template()` (lines 110‑150 in [`dataset_manager.py`](https://github.com/huggingface/skills/blob/main/dataset_manager.py)) to check every incoming row against the JSON schema defined in the selected template. If any row fails validation—such as missing required fields or incorrect data types—the script aborts and reports the specific validation error without uploading partial data to the Hub.

### Can I query existing public datasets and save subsets using this skill?

Yes, the [`sql_manager.py`](https://github.com/huggingface/skills/blob/main/sql_manager.py) script provides a DuckDB‑powered SQL interface for querying any public dataset on the Hugging Face Hub. You can run arbitrary SQL queries against datasets using the `hf://` protocol, then use `push_to_hub()` (lines 332‑365) to write the filtered results to a new repository. This workflow is useful for creating task‑specific subsets from large corpora like MMLU or for preprocessing data before fine‑tuning.

### How does the skill handle authentication with the Hugging Face Hub?

The skill relies on the standard `huggingface_hub` library for authentication, reusing the same HF token across both [`dataset_manager.py`](https://github.com/huggingface/skills/blob/main/dataset_manager.py) and [`sql_manager.py`](https://github.com/huggingface/skills/blob/main/sql_manager.py). When you run CLI commands, the underlying `HfApi` client automatically picks up your token from environment variables (`HF_TOKEN`) or local Hugging Face CLI configuration. This ensures consistent authentication whether you are creating repositories, uploading files, or querying datasets via SQL.