CSV vs JSON Profile Formats in Mirofish: Twitter and Reddit Agent Differences

Mirofish uses CSV for Twitter agents and JSON for Reddit agents, with Twitter requiring a flat 5-column structure while Reddit supports a hierarchical object model with optional demographic fields.

Mirofish leverages the OASIS simulation back-end to generate agent profiles for social media platforms. When the OasisProfileGenerator creates agent profiles, it must output them in the format required by each specific platform, resulting in fundamentally different file structures and schemas for Twitter versus Reddit agents.

Platform-Specific Format Requirements

The OASIS back-end expects different file formats based on the target social media platform's data model. This architectural decision reflects the original OASIS demo implementations, where Twitter uses simple tabular data while Reddit requires richer hierarchical structures.

Twitter CSV Structure

Twitter agents store profiles in twitter_profiles.csv as a flat, comma-separated file with exactly five columns. According to the source code in backend/app/services/oasis_profile_generator.py (lines 1065-1114), the _save_twitter_csv method writes a header row followed by sequential integer user_id values starting at 0.

The required CSV columns are:

• user_id: Sequential integer identifier starting at 0
• name: Display name of the agent
• username: Handle without the @ symbol
• user_char: Concatenation of bio and persona fields used internally by the LLM for character modeling
• description: Short, newline-stripped version of bio intended for public display

The user_char field combines the agent's biography and personality traits into a single string for the simulation engine, while description provides a cleaned version suitable for UI display.

Reddit JSON Structure

Reddit agents store profiles in reddit_profiles.json as a list of JSON objects with extensive demographic and behavioral fields. The _save_reddit_json method (lines 1141-1190) builds a Python list of dictionaries, each matching the structure returned by OasisAgentProfile.to_reddit_format().

Mandatory and default fields include:

• user_id: Guaranteed present (falls back to enumeration index if the model omits it)
• username, name, bio, persona: Core identity fields
• age: Defaults to 30 if unspecified
• gender: Normalized via _normalize_gender (lines 1166-1195), defaulting to "other"
• mbti: Defaults to "ISTJ" when not provided
• country: Defaults to "中国" (China) for unspecified locations
• karma, created_at, profession, interested_topics: Optional Reddit-specific metrics

The JSON output uses json.dump with ensure_ascii=False and indent=2 to preserve Unicode characters and maintain human-readable formatting.

Implementation in OasisProfileGenerator

The generator abstracts conversion logic into two private helper methods and a public dispatcher to keep the rest of the codebase agnostic to concrete file types.

The save_profiles(profiles, file_path, platform) method (lines 1048-1064) serves as the entry point. It routes to _save_twitter_csv when platform="twitter" and to _save_reddit_json when platform="reddit". This separation of concerns ensures that platform-specific serialization logic remains isolated within the profile generator service.

For Reddit profiles, the generator enforces data integrity by validating the presence of user_id and applying default values through _normalize_gender and other normalization helpers. The Twitter implementation focuses on string cleaning—specifically stripping newlines from the description field to prevent CSV parsing errors.

Reading Profiles in the Simulation API

The API endpoint that serves profiles mirrors the write-side logic to ensure the front-end receives uniform data shapes regardless of storage format. In backend/app/api/simulation.py (lines 84-90), the system checks the platform type and uses the appropriate parser:

if platform == "reddit":
    with open(profiles_file, "r", encoding="utf-8") as f:
        profiles = json.load(f)
else:
    with open(profiles_file, "r", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        profiles = list(reader)

This symmetric approach means the simulation engine receives Python dictionaries in both cases, though Reddit profiles contain nested demographic data while Twitter profiles remain flat key-value pairs.

Practical Examples

Generating Profiles in Both Formats

from backend.app.services.oasis_profile_generator import OasisProfileGenerator, OasisAgentProfile

profiles = [
    OasisAgentProfile(
        user_id=1,
        user_name="alice",
        name="Alice Zhang",
        bio="A passionate data scientist who loves cats.",
        persona="Analytical, curious, helpful.",
        age=28,
        gender="女",
        mbti="INTJ",
        country="China"
    ),
    OasisAgentProfile(
        user_id=2,
        user_name="bob",
        name="Bob Lee",
        bio="Tech blogger and indie game developer.",
        persona="Creative, outspoken, experimental."
    ),
]

gen = OasisProfileGenerator(api_key="dummy")

# Twitter output: flat CSV

gen.save_profiles(profiles, "twitter_profiles.csv", platform="twitter")

# Reddit output: hierarchical JSON

gen.save_profiles(profiles, "reddit_profiles.json", platform="reddit")

CSV Output (twitter_profiles.csv):

user_id,name,username,user_char,description
0,Alice Zhang,alice,A passionate data scientist who loves cats. Analytical, curious, helpful.,A passionate data scientist who loves cats.
1,Bob Lee,bob,Tech blogger and indie game developer.,Tech blogger and indie game developer.

JSON Output (reddit_profiles.json):

[
  {
    "user_id": 1,
    "username": "alice",
    "name": "Alice Zhang",
    "bio": "A passionate data scientist who loves cats.",
    "persona": "Analytical, curious, helpful.",
    "age": 28,
    "gender": "female",
    "mbti": "INTJ",
    "country": "China"
  },
  {
    "user_id": 2,
    "username": "bob",
    "name": "Bob Lee",
    "bio": "Tech blogger and indie game developer.",
    "persona": "Creative, outspoken, experimental.",
    "age": 30,
    "gender": "other",
    "mbti": "ISTJ",
    "country": "中国"
  }
]

Summary

• Twitter agents use a CSV format with five fixed columns (user_id, name, username, user_char, description) optimized for the OASIS simulation engine's csv.DictReader.
• Reddit agents use a JSON format containing rich hierarchical objects with demographic defaults (age 30, gender "other", MBTI "ISTJ") and mandatory user_id enforcement.
• The OasisProfileGenerator class encapsulates both formats through private helpers _save_twitter_csv and _save_reddit_json, exposed via the save_profiles dispatcher.
• The simulation API (backend/app/api/simulation.py) uses symmetric loading logic—json.load for Reddit and csv.DictReader for Twitter—to provide uniform Python dictionaries to the front-end.

Frequently Asked Questions

Why does Mirofish use different formats for Twitter and Reddit agents?

The format differences reflect the underlying OASIS back-end requirements and platform data models. Twitter agents require only basic identity fields suitable for a flat CSV structure that the original OASIS demo parses with csv.DictReader. Reddit agents require complex demographic data (MBTI types, karma scores, interested topics) that naturally fits JSON's hierarchical object model.

What happens if a Reddit profile is missing required fields like age or gender?

The _save_reddit_json method in backend/app/services/oasis_profile_generator.py applies automatic defaults. If age is missing, it defaults to 30. If gender is missing or set to "other", the _normalize_gender helper normalizes the value. MBTI defaults to "ISTJ" and country defaults to "中国" (China) when unspecified.

Can I manually edit the generated CSV or JSON files?

Yes, both formats are human-readable and editable. For twitter_profiles.csv, maintain the five-column header structure and ensure user_id values remain sequential integers starting at 0. For reddit_profiles.json, preserve the list-of-objects structure and ensure every object contains a user_id field, as the simulation API validates its presence when loading profiles.

How does the simulation API handle format validation?

The API endpoint in backend/app/api/simulation.py (lines 84-90) checks the platform parameter and uses the appropriate parser. It loads JSON directly for Reddit and CSV via DictReader for Twitter. While the reader performs basic format validation (ensuring CSV headers match or JSON is well-formed), the OASIS engine performs additional validation on required fields like user_id during simulation initialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}