Preparing Training Data for Llama 3 in Firefly Format: A Complete Guide

The llama3-chinese-chat repository provides a lightweight Python pipeline in tools/convert_raw_data_for_firefly.py that converts raw JSONL instruction datasets into Firefly format for Chinese Llama 3 fine-tuning.

Preparing training data for Llama3 in Firefly format is the first step toward instruction-tuning Chinese variants of Meta's Llama 3 model. The crazyboym/llama3-chinese-chat repository streamlines this process with dedicated conversion utilities that handle the schema transformation without external dependencies. These tools bridge the gap between raw instruction datasets and the Firefly format used by popular Chinese fine-tuning frameworks.

Understanding the Firefly Format Structure

The Firefly format wraps conversational data in a specific JSON schema designed for instruction tuning. Each training example becomes a JSON object containing a conversation key, which holds a list of turn-based interactions between human and assistant roles.

According to the source code, a valid Firefly entry follows this structure:

{"conversation": [{"human": "instruction + input text", "assistant": "output text"}]}

This differs from the raw input format by consolidating the instruction and optional input fields into a single human prompt, while preserving the assistant's response unchanged.

Converting Raw JSONL to Firefly Format

The primary conversion utility lives in tools/convert_raw_data_for_firefly.py. This script uses only Python's standard library to process line-oriented JSON files, making it portable across different training pipelines.

Required Input Schema

Before conversion, your raw JSONL file must contain lines with these keys:

• instruction (required): The task description or command
• input (optional): Additional context or content to process
• output (required): The target response or answer

Example raw entry:

{"instruction": "翻译以下句子", "input": "我爱学习", "output": "I love studying."}

The Three-Step Conversion Process

The tools/convert_raw_data_for_firefly.py script executes the following transformation as implemented in the repository:

1. Load raw entries – Parses each line of the input JSONL file into a Python dictionary
2. Merge instruction and input – Concatenates the instruction and input values to form the human prompt, leaving the assistant's output unchanged
3. Wrap in Firefly schema – Packages the pair into {"conversation": [{"human": ..., "assistant": ...}]} and serializes to the output file

Command-Line Usage

Run the converter from the repository root:

python tools/convert_raw_data_for_firefly.py \
    --input_file raw.jsonl \
    --output_file firefly.jsonl

This produces firefly.jsonl with Firefly-compatible lines:

{"conversation": [{"human": "翻译以下句子我爱学习", "assistant": "I love studying."}]}

Converting Firefly to ShareGPT Format

Many modern fine-tuning frameworks like LLaMA-Factory and XTuner expect the ShareGPT schema instead of Firefly. The repository includes tools/convert_firefly_data_to_sharegpt.py to handle this secondary conversion without rewriting your preprocessing pipeline.

To convert an existing Firefly file to ShareGPT format:

python tools/convert_firefly_data_to_sharegpt.py \
    --input_file firefly.jsonl \
    --output_file sharegpt.jsonl

The resulting ShareGPT structure uses role-based from fields:

{"conversations": [{"from": "human", "value": "翻译以下句子我爱学习"}, {"from": "gpt", "value": "I love studying."}]}

Programmatic Integration

For custom data preprocessing workflows, import the conversion functions directly into Python scripts rather than using the CLI:

from tools.convert_raw_data_for_firefly import convert_jsonl

# Convert in-process without shell execution

convert_jsonl("raw.jsonl", "firefly.jsonl")

This approach allows you to chain the Firefly conversion with other data cleaning steps, such as filtering by length or deduplication, before feeding the results into training frameworks.

Summary

• Primary converter: tools/convert_raw_data_for_firefly.py transforms raw instruction JSONL into Firefly format by merging instruction and input fields
• Input requirements: Each line needs instruction, optional input, and output keys
• Output schema: Firefly format wraps conversations in {"conversation": [{"human": "...", "assistant": "..."}]}
• Secondary converter: tools/convert_firefly_data_to_sharegpt.py bridges Firefly to ShareGPT for compatibility with LLaMA-Factory and similar tools
• Zero dependencies: Both scripts use only Python standard library modules

Frequently Asked Questions

What is the Firefly format used for in Llama 3 training?

The Firefly format is a JSON schema designed for instruction-tuning large language models on Chinese conversational data. It structures each training example as a conversation array containing human prompts and assistant responses, which aligns with the training objectives of the Firefly dataset popular in Chinese NLP communities.

Can I use these converters with the LLaMA-Factory framework?

Yes, but you need an intermediate step. First convert your raw data to Firefly format using tools/convert_raw_data_for_firefly.py, then use tools/convert_firefly_data_to_sharegpt.py to produce ShareGPT-compatible JSONL. LLaMA-Factory natively supports the ShareGPT schema, making this two-step conversion the recommended path for integration.

Do I need to install additional Python packages to run the converters?

No. Both tools/convert_raw_data_for_firefly.py and tools/convert_firefly_data_to_sharegpt.py rely exclusively on Python's standard library (primarily the json module). This design choice ensures compatibility across different environments without dependency management overhead.

How do I handle raw data that lacks the input field?

The conversion script treats the input field as optional. When input is missing or empty, the converter uses only the instruction field for the human prompt. This flexibility allows you to process both pure instruction datasets and context-dependent tasks through the same pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}