# Data Schema for Storing Job Postings in ai-job-search: JSON, CSV, and Raw Text Architecture

> Discover the data schema for storing job postings in ai-job-search. Learn how raw text, JSON, and CSV architecture enable machine processing and human readability for efficient job data management.

- Repository: [Mads Lorentzen/ai-job-search](https://github.com/MadsLorentzen/ai-job-search)
- Tags: architecture
- Published: 2026-07-25

---

**The ai-job-search framework stores job postings in a three-layer architecture combining raw text files, a deduplicated JSON index, and a CSV application tracker to ensure both human readability and machine processing.**

Managing job search data requires a flexible yet structured approach. The ai-job-search repository implements a specific data schema for storing job postings that preserves original content while enabling automated deduplication and application tracking. This design separates concerns across three distinct storage layers, each optimized for different workflows from initial discovery to final application.

## Three-Layer Storage Architecture

The repository follows a **thin-pointer** principle where raw text lives once in the `documents/` folder while automated agents work against lightweight JSON views.

### Raw Text Layer (Human-Readable)

Original job posting content lives in `documents/postings/<Company> - <Job Title>.txt` as defined in [[`documents/README.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/documents/README.md)](documents/README.md). This layer maintains no enforced schema beyond the filename convention—the entire file content is treated as a single string that downstream parsers process. The human-readable format ensures you can always review the original posting text even after automated processing.

### JSON Index Layer (Machine-Readable Deduplication)

The scraper maintains [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json) to prevent processing duplicates. According to [[`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md)](.claude/skills/job-scraper/SKILL.md), each entry follows this strict schema:

```json
{
  "title": "Job Title",
  "company": "Company Name",
  "url": "https://example.com/job/123",
  "first_seen": "2024-01-15",
  "fit": "high|medium|low",
  "status": "new|skipped|evaluated|ranked|expired",
  "portal": "source-portal-name"
}

```

The system generates composite keys using `{company}_{title}` (lowercased with spaces converted to underscores) to uniquely identify postings regardless of URL variations. This guarantees that duplicate URLs or company-title combinations are ignored during subsequent scrape operations.

### CSV Tracker Layer (Application History)

Permanent application records reside in `job_search_tracker.csv` at the repository root. As documented in [[`.claude/commands/outcome.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/commands/outcome.md)](.claude/commands/outcome.md), this file uses stable column ordering with fields including `company`, `role`, `source`, `status`, and `date_applied`. New columns are added additively to preserve backward compatibility, making this CSV the source of truth for the `/html-report` and `/upskill` commands.

## How the Storage Layers Interact

The framework orchestrates data flow through three distinct operations:

1. **Ingestion**: Add postings by dropping text files into `documents/postings/` or submitting URLs via the `/apply` command documented in [[`.claude/commands/apply.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/commands/apply.md)](.claude/commands/apply.md)

2. **Indexing**: When `/scrape` runs, it reads portal CLI outputs, fetches full postings via `WebFetch` when needed, and writes structured JSON entries to [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json)

3. **Application**: Upon applying, the framework copies posting text to `documents/applications/<company>_<role>/job_posting.md` while appending metadata to `job_search_tracker.csv`

Agents preserve unknown keys when rewriting [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json), allowing schema evolution without migration scripts.

## Programmatically Working with the Schema

To manually add entries to the JSON index:

```python
import json
from pathlib import Path
import datetime

def add_seen_job(url: str, title: str, company: str, portal: str, fit: str):
    path = Path("job_scraper/seen_jobs.json")
    data = {"seen": {}} if not path.exists() else json.loads(path.read_text())
    
    key = f"{company}_{title}".replace(" ", "_").lower()
    data["seen"][key] = {
        "title": title,
        "company": company,
        "url": url,
        "first_seen": datetime.date.today().isoformat(),
        "fit": fit,
        "status": "new",
        "portal": portal,
    }
    path.write_text(json.dumps(data, indent=2))

# Example usage

add_seen_job(
    url="https://jobindex.dk/job/1234567",
    title="AI Engineer",
    company="Acme Corp",
    portal="jobindex-search",
    fit="high",
)

```

For manual workflow integration:

```bash

# Create raw posting storage

mkdir -p documents/postings
cat <<EOF > documents/postings/Acme\ Corp\ -\ AI\ Engineer.txt
[Full job posting content here]
EOF

# Trigger indexing

/scrape

```

## Summary

- The **data schema for storing job postings** spans three layers: unstructured text in `documents/postings/`, structured JSON in [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json), and tabular history in `job_search_tracker.csv`
- The JSON schema enforces required fields for `title`, `company`, `url`, `first_seen`, `fit` (high/medium/low), `status` (new/skipped/evaluated/ranked/expired), and `portal`
- Composite keys prevent duplicates based on company-title combinations rather than URLs alone
- The CSV tracker maintains additive schema evolution, ensuring reporting commands continue functioning as new tracking fields are introduced
- Raw source text remains available in `documents/` while agents work against lightweight JSON pointers

## Frequently Asked Questions

### What happens if I add custom fields to the JSON schema?

The framework handles schema evolution gracefully. When agents rewrite [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json), they preserve unknown keys, allowing you to add fields like `salary_range` or `remote_policy` without breaking existing functionality or requiring migration scripts.

### Can I store job postings without using the automated scraper?

Yes. Simply create text files in `documents/postings/` following the `<Company> - <Job Title>.txt` naming convention. While these won't appear in the JSON index until processed by `/scrape` or `/apply`, the system treats them as valid raw postings for manual review.

### How does the system prevent duplicate job postings?

The JSON index generates unique keys using the pattern `{company}_{title}` with spaces replaced by underscores and text normalized to lowercase. When the scraper encounters an existing company-title combination, it skips creating a new entry regardless of whether the URL differs from previous sightings.

### Where is application history permanently stored?

Application history persists in `job_search_tracker.csv` at the repository root. This file serves as the authoritative source for the `/outcome`, `/upskill`, and `/html-report` commands, tracking every position you've applied to along with dates and outcomes.