Creating and Managing Hugging Face Hub Datasets with System Prompts

The hugging-face-datasets skill in the huggingface/skills repository enables you to create, configure, and populate Hugging Face Hub datasets while embedding system prompts directly into repository configuration files using JSON templates.

Managing machine learning datasets requires consistent schema enforcement and embedded instructions for data generation or model fine-tuning. The huggingface/skills repository provides a dedicated hugging-face-datasets skill that streamlines creating and managing Hugging Face Hub datasets with system prompts, allowing you to initialize repositories, validate row structures against templates, and persist system-level instructions in version-controlled config.json files.

Architecture of the Hugging Face Datasets Skill

The skill is organized into three logical layers that handle command parsing, Hub API interactions, and template management.

CLI Entry Point and Command Routing

The command-line interface is implemented in skills/hugging-face-datasets/scripts/dataset_manager.py. The if __name__ == "__main__" block (lines 25‑50) constructs an argparse hierarchy that exposes subcommands including init, quick_setup, add_rows, and stats. When you invoke a command, the parser forwards arguments to the appropriate library function, such as quick_setup() for automated repository initialization.

Dataset Manager Core Functions

The core logic for Hub interactions resides in functions like init_dataset(), define_config(), and add_rows() within the same file. These wrappers around the huggingface_hub library handle repository creation, metadata upload, and JSON‑Lines streaming. For example, define_config() (lines 74‑95) writes the config.json file that stores dataset metadata and system prompts, while add_rows() (lines 55‑63) validates incoming data against the template schema before appending to the dataset.

Template System for System Prompts

JSON templates defining common data shapes—such as chat, classification, and QA—are stored in skills/hugging-face-datasets/templates/*.json. Each template includes a system_prompt field that describes the intended AI behavior for data generated under that schema. When you select a template via the --template argument, the skill automatically extracts this prompt and persists it in the repository configuration.

How System Prompts Are Embedded

System prompts are embedded into datasets through a structured workflow that bridges template selection and configuration persistence.

Selecting and Loading Templates

When you run quick_setup with the --template flag, the skill invokes load_dataset_template() (line 97) to read the corresponding JSON file from the templates directory. For instance, selecting --template chat loads templates/chat.json, which contains a predefined schema and a system_prompt value such as "You are an AI assistant …".

Persisting Prompts in config.json

After loading the template, quick_setup() extracts the system prompt using template_config.get("system_prompt", "") (line 38) and passes it to define_config(). This function writes a config.json file (lines 84‑90) that includes a "system_prompt" key alongside other metadata. Downstream tools—such as fine‑tuning scripts or evaluation pipelines—can read this file to ensure generated data adheres to the described behavior.

Practical Workflow: Creating and Managing Datasets

The skill supports a complete lifecycle for dataset management, from repository creation to data ingestion and transformation.

Initialize a Dataset with quick_setup

The fastest way to create a new dataset with an embedded system prompt is using the quick_setup subcommand:

uv run skills/hugging-face-datasets/scripts/dataset_manager.py quick_setup \
    --repo_id "myuser/my-chat-dataset" \
    --template chat

This command creates the repository myuser/my-chat-dataset on the Hugging Face Hub, uploads a README and config.json containing the chat system prompt from templates/chat.json, and optionally seeds the dataset with example rows if the template includes them.

Validate and Stream Rows

To append new data while enforcing the template schema, use the add_rows subcommand:

# Prepare a JSON‑Lines file with new examples

cat <<EOF > new_rows.jsonl
{"messages":[{"role":"user","content":"How do I bake a cake?"},{"role":"assistant","content":"Here is a recipe …"}]}
EOF

# Stream validated rows into the dataset

uv run skills/hugging-face-datasets/scripts/dataset_manager.py add_rows \
    --repo_id "myuser/my-chat-dataset" \
    --split train \
    --template chat \
    --rows_json "$(cat new_rows.jsonl)"

The add_rows function calls validate_by_template() (lines 110‑150) to ensure every row conforms to the chat schema before uploading. If validation fails, the script aborts and reports the specific error without uploading partial data to the Hub.

Query and Transform with SQL Manager

For advanced workflows, the sql_manager.py script enables DuckDB‑powered querying and transformation of existing datasets:

uv run skills/hugging-face-datasets/scripts/sql_manager.py query \
    --dataset "cais/mmlu" \
    --sql "SELECT * FROM data WHERE subject='nutrition'" \
    --push-to "myuser/nutrition-subset"

This command builds an hf:// protocol path, executes the SQL query, and invokes push_to_hub() (lines 332‑365) to write the filtered results to a new repository. The SQL manager reuses the same Hugging Face token and Hub API patterns as the dataset manager, ensuring consistent authentication across the skill suite.

Programmatic Python API

You can also embed dataset creation and system prompt management directly into Python training pipelines:

from skills.hugging-face-datasets.scripts.dataset_manager import define_config
import json
import pathlib

# Define target repository

repo_id = "myuser/my-chat-dataset"

# Load system prompt from template

template_path = pathlib.Path(__file__).parent.parent / "templates" / "chat.json"
prompt = json.loads(template_path.read_text())["system_prompt"]

# Persist configuration with embedded system prompt

define_config(repo_id, system_prompt=prompt)

This approach allows fine‑tuning frameworks to dynamically read the system_prompt from config.json and generate training examples that adhere to the specified behavior.

Summary

• The hugging-face-datasets skill provides a CLI and Python API for creating and managing Hugging Face Hub datasets with system prompts.
• System prompts are embedded via JSON templates (templates/*.json) and persisted in repository config.json files through define_config().
• The dataset_manager.py script handles repository initialization (quick_setup), row validation (validate_by_template at lines 110‑150), and data streaming (add_rows).
• The sql_manager.py script enables DuckDB‑based querying and subset creation via push_to_hub() (lines 332‑365).
• All workflows leverage the huggingface_hub API and reuse authentication tokens across the skill suite.

Frequently Asked Questions

How do I embed a custom system prompt when creating a new dataset?

You can embed a custom system prompt by using the quick_setup command with a template that contains your desired prompt, or by manually calling define_config() in Python. The skill stores the prompt in config.json at the repository root, making it accessible to downstream training scripts. If you need a completely custom prompt, create a new JSON file in the templates/ directory with a "system_prompt" key and pass it to --template.

What validation does the skill perform when adding rows to a dataset?

When you use the add_rows subcommand or call the function directly, the skill invokes validate_by_template() (lines 110‑150 in dataset_manager.py) to check every incoming row against the JSON schema defined in the selected template. If any row fails validation—such as missing required fields or incorrect data types—the script aborts and reports the specific validation error without uploading partial data to the Hub.

Can I query existing public datasets and save subsets using this skill?

Yes, the sql_manager.py script provides a DuckDB‑powered SQL interface for querying any public dataset on the Hugging Face Hub. You can run arbitrary SQL queries against datasets using the hf:// protocol, then use push_to_hub() (lines 332‑365) to write the filtered results to a new repository. This workflow is useful for creating task‑specific subsets from large corpora like MMLU or for preprocessing data before fine‑tuning.

How does the skill handle authentication with the Hugging Face Hub?

The skill relies on the standard huggingface_hub library for authentication, reusing the same HF token across both dataset_manager.py and sql_manager.py. When you run CLI commands, the underlying HfApi client automatically picks up your token from environment variables (HF_TOKEN) or local Hugging Face CLI configuration. This ensures consistent authentication whether you are creating repositories, uploading files, or querying datasets via SQL.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}