# CSV vs JSON Profile Formats in Mirofish: Twitter and Reddit Agent Differences

> Explore CSV vs JSON profile formats in Mirofish for Twitter and Reddit agents. Understand key differences in data structure and field requirements for efficient agent management.

- Repository: [BaiFu/mirofish](https://github.com/666ghj/mirofish)
- Tags: deep-dive
- Published: 2026-02-23

---

**Mirofish uses CSV for Twitter agents and JSON for Reddit agents, with Twitter requiring a flat 5-column structure while Reddit supports a hierarchical object model with optional demographic fields.**

Mirofish leverages the OASIS simulation back-end to generate agent profiles for social media platforms. When the `OasisProfileGenerator` creates agent profiles, it must output them in the format required by each specific platform, resulting in fundamentally different file structures and schemas for Twitter versus Reddit agents.

## Platform-Specific Format Requirements

The OASIS back-end expects different file formats based on the target social media platform's data model. This architectural decision reflects the original OASIS demo implementations, where Twitter uses simple tabular data while Reddit requires richer hierarchical structures.

### Twitter CSV Structure

Twitter agents store profiles in `twitter_profiles.csv` as a flat, comma-separated file with exactly five columns. According to the source code in [`backend/app/services/oasis_profile_generator.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/services/oasis_profile_generator.py) (lines 1065-1114), the `_save_twitter_csv` method writes a header row followed by sequential integer `user_id` values starting at 0.

The required CSV columns are:

- **user_id**: Sequential integer identifier starting at 0
- **name**: Display name of the agent
- **username**: Handle without the @ symbol
- **user_char**: Concatenation of `bio` and `persona` fields used internally by the LLM for character modeling
- **description**: Short, newline-stripped version of `bio` intended for public display

The `user_char` field combines the agent's biography and personality traits into a single string for the simulation engine, while `description` provides a cleaned version suitable for UI display.

### Reddit JSON Structure

Reddit agents store profiles in [`reddit_profiles.json`](https://github.com/666ghj/mirofish/blob/main/reddit_profiles.json) as a list of JSON objects with extensive demographic and behavioral fields. The `_save_reddit_json` method (lines 1141-1190) builds a Python list of dictionaries, each matching the structure returned by `OasisAgentProfile.to_reddit_format()`.

Mandatory and default fields include:

- **user_id**: Guaranteed present (falls back to enumeration index if the model omits it)
- **username**, **name**, **bio**, **persona**: Core identity fields
- **age**: Defaults to 30 if unspecified
- **gender**: Normalized via `_normalize_gender` (lines 1166-1195), defaulting to "other"
- **mbti**: Defaults to "ISTJ" when not provided
- **country**: Defaults to "中国" (China) for unspecified locations
- **karma**, **created_at**, **profession**, **interested_topics**: Optional Reddit-specific metrics

The JSON output uses `json.dump` with `ensure_ascii=False` and `indent=2` to preserve Unicode characters and maintain human-readable formatting.

## Implementation in OasisProfileGenerator

The generator abstracts conversion logic into two private helper methods and a public dispatcher to keep the rest of the codebase agnostic to concrete file types.

The `save_profiles(profiles, file_path, platform)` method (lines 1048-1064) serves as the entry point. It routes to `_save_twitter_csv` when `platform="twitter"` and to `_save_reddit_json` when `platform="reddit"`. This separation of concerns ensures that platform-specific serialization logic remains isolated within the profile generator service.

For Reddit profiles, the generator enforces data integrity by validating the presence of `user_id` and applying default values through `_normalize_gender` and other normalization helpers. The Twitter implementation focuses on string cleaning—specifically stripping newlines from the description field to prevent CSV parsing errors.

## Reading Profiles in the Simulation API

The API endpoint that serves profiles mirrors the write-side logic to ensure the front-end receives uniform data shapes regardless of storage format. In [`backend/app/api/simulation.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/api/simulation.py) (lines 84-90), the system checks the platform type and uses the appropriate parser:

```python
if platform == "reddit":
    with open(profiles_file, "r", encoding="utf-8") as f:
        profiles = json.load(f)
else:
    with open(profiles_file, "r", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        profiles = list(reader)

```

This symmetric approach means the simulation engine receives Python dictionaries in both cases, though Reddit profiles contain nested demographic data while Twitter profiles remain flat key-value pairs.

## Practical Examples

### Generating Profiles in Both Formats

```python
from backend.app.services.oasis_profile_generator import OasisProfileGenerator, OasisAgentProfile

profiles = [
    OasisAgentProfile(
        user_id=1,
        user_name="alice",
        name="Alice Zhang",
        bio="A passionate data scientist who loves cats.",
        persona="Analytical, curious, helpful.",
        age=28,
        gender="女",
        mbti="INTJ",
        country="China"
    ),
    OasisAgentProfile(
        user_id=2,
        user_name="bob",
        name="Bob Lee",
        bio="Tech blogger and indie game developer.",
        persona="Creative, outspoken, experimental."
    ),
]

gen = OasisProfileGenerator(api_key="dummy")

# Twitter output: flat CSV

gen.save_profiles(profiles, "twitter_profiles.csv", platform="twitter")

# Reddit output: hierarchical JSON

gen.save_profiles(profiles, "reddit_profiles.json", platform="reddit")

```

**CSV Output (`twitter_profiles.csv`):**

```csv
user_id,name,username,user_char,description
0,Alice Zhang,alice,A passionate data scientist who loves cats. Analytical, curious, helpful.,A passionate data scientist who loves cats.
1,Bob Lee,bob,Tech blogger and indie game developer.,Tech blogger and indie game developer.

```

**JSON Output ([`reddit_profiles.json`](https://github.com/666ghj/mirofish/blob/main/reddit_profiles.json)):**

```json
[
  {
    "user_id": 1,
    "username": "alice",
    "name": "Alice Zhang",
    "bio": "A passionate data scientist who loves cats.",
    "persona": "Analytical, curious, helpful.",
    "age": 28,
    "gender": "female",
    "mbti": "INTJ",
    "country": "China"
  },
  {
    "user_id": 2,
    "username": "bob",
    "name": "Bob Lee",
    "bio": "Tech blogger and indie game developer.",
    "persona": "Creative, outspoken, experimental.",
    "age": 30,
    "gender": "other",
    "mbti": "ISTJ",
    "country": "中国"
  }
]

```

## Summary

- **Twitter agents** use a **CSV format** with five fixed columns (`user_id`, `name`, `username`, `user_char`, `description`) optimized for the OASIS simulation engine's `csv.DictReader`.
- **Reddit agents** use a **JSON format** containing rich hierarchical objects with demographic defaults (age 30, gender "other", MBTI "ISTJ") and mandatory `user_id` enforcement.
- The `OasisProfileGenerator` class encapsulates both formats through private helpers `_save_twitter_csv` and `_save_reddit_json`, exposed via the `save_profiles` dispatcher.
- The simulation API ([`backend/app/api/simulation.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/api/simulation.py)) uses symmetric loading logic—`json.load` for Reddit and `csv.DictReader` for Twitter—to provide uniform Python dictionaries to the front-end.

## Frequently Asked Questions

### Why does Mirofish use different formats for Twitter and Reddit agents?

The format differences reflect the underlying OASIS back-end requirements and platform data models. Twitter agents require only basic identity fields suitable for a flat CSV structure that the original OASIS demo parses with `csv.DictReader`. Reddit agents require complex demographic data (MBTI types, karma scores, interested topics) that naturally fits JSON's hierarchical object model.

### What happens if a Reddit profile is missing required fields like age or gender?

The `_save_reddit_json` method in [`backend/app/services/oasis_profile_generator.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/services/oasis_profile_generator.py) applies automatic defaults. If `age` is missing, it defaults to 30. If `gender` is missing or set to "other", the `_normalize_gender` helper normalizes the value. MBTI defaults to "ISTJ" and country defaults to "中国" (China) when unspecified.

### Can I manually edit the generated CSV or JSON files?

Yes, both formats are human-readable and editable. For `twitter_profiles.csv`, maintain the five-column header structure and ensure `user_id` values remain sequential integers starting at 0. For [`reddit_profiles.json`](https://github.com/666ghj/mirofish/blob/main/reddit_profiles.json), preserve the list-of-objects structure and ensure every object contains a `user_id` field, as the simulation API validates its presence when loading profiles.

### How does the simulation API handle format validation?

The API endpoint in [`backend/app/api/simulation.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/api/simulation.py) (lines 84-90) checks the platform parameter and uses the appropriate parser. It loads JSON directly for Reddit and CSV via `DictReader` for Twitter. While the reader performs basic format validation (ensuring CSV headers match or JSON is well-formed), the OASIS engine performs additional validation on required fields like `user_id` during simulation initialization.