Preparing Training Data for Llama 3 in ShareGPT Format: Step-by-Step Guide

To prepare training data for Llama 3 in ShareGPT format, convert raw JSONL datasets to the Firefly schema using tools/convert_raw_data_for_firefly.py, then transform them to ShareGPT structure with tools/convert_firefly_data_to_sharegpt.py.

The crazyboym/llama3-chinese-chat repository provides a complete data preparation pipeline for fine-tuning Llama 3 on Chinese conversational datasets. Preparing training data for Llama 3 in ShareGPT format requires transforming raw instruction-following datasets through a two-stage conversion process before they can be consumed by frameworks like LLaMA-Factory or Firefly.

Repository Architecture for Data Preparation

The data preparation submodule follows a layered pipeline architecture designed to normalize diverse raw formats into the standard ShareGPT schema.

Layer	Description	Key Files
Raw Data Ingestion	Parses line-wise JSONL files containing fields like instruction, input, and output.	tools/convert_raw_data_for_firefly.py
Schema Normalization	Wraps entries into a single-element conversation list with human and assistant roles.	tools/convert_raw_data_for_firefly.py
ShareGPT Conversion	Maps normalized conversations to ShareGPT format using from and value keys.	tools/convert_firefly_data_to_sharegpt.py
Quality Utilities	Provides sampling, token counting, and JSONL validation helpers.	tools/sample_data.py, tools/count_data.py, tools/check_jsonl.py

All conversion scripts reside in /tools, while inference demos and deployment utilities live under /deploy.

Converting Raw Data to Firefly Schema

The first step transforms arbitrary instruction-format JSONL into the Firefly-style conversation structure. In tools/convert_raw_data_for_firefly.py, the convert_entry() function builds human-assistant pairs from standard instruction fields.

# tools/convert_raw_data_for_firefly.py

import json

def convert_entry(entry):
    # Combine instruction and input for the human turn

    return {
        "human": entry["instruction"] + entry["input"],
        "assistant": entry["output"]
    }

def convert_jsonl(input_file, output_file):
    with open(input_file, "r", encoding="utf-8") as f_in, \
         open(output_file, "w", encoding="utf-8") as f_out:
        for line in f_in:
            entry = json.loads(line)
            conv = {"conversation": [convert_entry(entry)]}
            json.dump(conv, f_out, ensure_ascii=False)
            f_out.write("\n")

Execute the conversion from the repository root:

python tools/convert_raw_data_for_firefly.py \
    --input ./m-a-p/COIG-CQIA/zhihu/zhihu_expansion.jsonl \
    --output zhihu_expansion_firefly.jsonl

This produces an intermediate file where each line contains a conversation list with human and assistant keys.

Transforming Firefly to ShareGPT Format

The second step converts the Firefly schema to ShareGPT format using tools/convert_firefly_data_to_sharegpt.py. This script maps role names to ShareGPT identifiers (human → human, assistant → gpt) and restructures the data to use from and value fields.

# tools/convert_firefly_data_to_sharegpt.py

import json

def convert_jsonl(input_file, output_file):
    with open(input_file, "r", encoding="utf-8") as fin, \
         open(output_file, "w", encoding="utf-8") as fout:
        for line in fin:
            data = json.loads(line.strip())
            convs = data["conversation"]
            new_convs = []
            for conv in convs:
                for role, txt in conv.items():
                    # Map roles to ShareGPT standard

                    new_role = "gpt" if role == "assistant" else "human"
                    new_convs.append({"from": new_role, "value": txt})
            fout.write(json.dumps({"conversations": new_convs},
                                 ensure_ascii=False) + "\n")

Run the transformation:

python tools/convert_firefly_data_to_sharegpt.py \
    --input zhihu_expansion_firefly.jsonl \
    --output zhihu_expansion_sharegpt.jsonl

The resulting ShareGPT records follow this schema:

{
  "conversations": [
    {"from": "human", "value": "你好，请介绍一下中国古诗词。"},
    {"from": "gpt", "value": "中国古诗词历史悠久..."}
  ]
}

Data Validation and Sampling Utilities

Before training, use the utility scripts to validate and inspect your dataset.

Sampling a subset with tools/sample_data.py:

# tools/sample_data.py

import random

def sample_jsonl(input_file, output_file, ratio=0.33):
    with open(input_file, "r", encoding="utf-8") as f:
        lines = f.readlines()
    sample = random.sample(lines, int(len(lines) * ratio))
    with open(output_file, "w", encoding="utf-8") as f:
        f.writelines(sample)

Counting token statistics with tools/count_data.py:

# tools/count_data.py

def count_jsonl(input_file):
    max_len = min_len = total = cnt = 0
    with open(input_file, "r", encoding="utf-8") as f:
        for line in f:
            length = len(line.strip().split())
            max_len = max(max_len, length)
            min_len = min(min_len, length) if cnt else length
            total += length
            cnt += 1
    print(f"max: {max_len}, min: {min_len}, avg: {total / cnt:.2f}, lines: {cnt}")

Integrating with Fine-Tuning Frameworks

Once your data is in ShareGPT format, it is compatible with major fine-tuning frameworks. Both LLaMA-Factory and Firefly accept JSONL files where each line contains a conversations array with from and value keys.

Example LLaMA-Factory training command:

llama_factory train \
    --model_name_or_path meta-llama/Meta-Llama-3-8B \
    --train_file zhihu_expansion_sharegpt.jsonl \
    --output_dir output/llama3_zhihu \
    --lora_r 128 --lora_alpha 256 \
    --epochs 3

Inference and Deployment

After fine-tuning, test your model using the provided inference demos. The repository includes deploy/python/chat_demo.py for CLI interaction and deploy/streamlit/web_llama3_chat.py for web-based testing.

Run the CLI demo:

python deploy/python/chat_demo.py \
    --model_name_or_path shareAI/llama3-Chinese-chat-8b \
    --load_in_4bit False

For 4-bit quantized inference, set --load_in_4bit True and configure the BitsAndBytesConfig in the script.

Launch the Streamlit web interface:

streamlit run deploy/streamlit/web_llama3_chat.py \
    --theme.base="dark" \
    /path/to/llama3-chinese-chat-8b

Best Practices for ShareGPT Data Preparation

• Validate JSONL structure before training by running python -m json.tool on sample lines or using tools/check_jsonl.py to catch malformed entries early.
• Maintain flat conversation lists — ShareGPT expects a linear sequence of turns. Nested conversation objects will cause training failures in most frameworks.
• Sample small subsets first using tools/sample_data.py to verify data loading before committing to full training runs.
• Use 4-bit quantization for inference on limited GPU memory by enabling load_in_4bit=True in chat_demo.py.
• Preserve schema consistency by version-controlling any modifications to the conversion scripts in tools/.

Summary

• Two-stage conversion is required: raw JSONL → Firefly schema (tools/convert_raw_data_for_firefly.py) → ShareGPT format (tools/convert_firefly_data_to_sharegpt.py).
• ShareGPT schema requires conversations arrays containing objects with from (role) and value (text) fields.
• Utility scripts (sample_data.py, count_data.py) help validate dataset size and distribution before training.
• Framework compatibility extends to LLaMA-Factory, Firefly, and SWIFT once data is in ShareGPT format.
• Inference tools include a Python CLI demo and Streamlit web interface under the /deploy directory.

Frequently Asked Questions

What is the ShareGPT format for Llama 3 training?

The ShareGPT format structures conversations as a JSON object with a conversations key containing an array of turns. Each turn has a from field identifying the speaker (human or gpt) and a value field containing the message text. According to the crazyboym/llama3-chinese-chat source code, this schema is produced by tools/convert_firefly_data_to_sharegpt.py and is compatible with major fine-tuning frameworks like LLaMA-Factory.

Can I convert existing instruction datasets to ShareGPT format?

Yes. If your dataset uses standard instruction fields (instruction, input, output), use tools/convert_raw_data_for_firefly.py to create an intermediate Firefly schema, then run tools/convert_firefly_data_to_sharegpt.py to produce the final ShareGPT file. This pipeline handles the role mapping and structural normalization automatically.

How do I verify my ShareGPT file is correctly formatted?

Use tools/check_jsonl.py to validate JSON structure, or run python tools/count_data.py your_file.jsonl to check line counts and token statistics. Additionally, inspect a few samples manually to ensure the conversations array contains alternating human and gpt entries with valid from and value keys.

Which fine-tuning frameworks support this ShareGPT format?

The ShareGPT format produced by this repository is compatible with LLaMA-Factory, Firefly, and SWIFT. These frameworks expect the conversations array structure and automatically handle the from/value schema during tokenization and training.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}