Data Schema for Storing Job Postings in ai-job-search: JSON, CSV, and Raw Text Architecture

The ai-job-search framework stores job postings in a three-layer architecture combining raw text files, a deduplicated JSON index, and a CSV application tracker to ensure both human readability and machine processing.

Managing job search data requires a flexible yet structured approach. The ai-job-search repository implements a specific data schema for storing job postings that preserves original content while enabling automated deduplication and application tracking. This design separates concerns across three distinct storage layers, each optimized for different workflows from initial discovery to final application.

Three-Layer Storage Architecture

The repository follows a thin-pointer principle where raw text lives once in the documents/ folder while automated agents work against lightweight JSON views.

Raw Text Layer (Human-Readable)

Original job posting content lives in documents/postings/<Company> - <Job Title>.txt as defined in [documents/README.md](documents/README.md). This layer maintains no enforced schema beyond the filename convention—the entire file content is treated as a single string that downstream parsers process. The human-readable format ensures you can always review the original posting text even after automated processing.

JSON Index Layer (Machine-Readable Deduplication)

The scraper maintains job_scraper/seen_jobs.json to prevent processing duplicates. According to [.claude/skills/job-scraper/SKILL.md](.claude/skills/job-scraper/SKILL.md), each entry follows this strict schema:

{
  "title": "Job Title",
  "company": "Company Name",
  "url": "https://example.com/job/123",
  "first_seen": "2024-01-15",
  "fit": "high|medium|low",
  "status": "new|skipped|evaluated|ranked|expired",
  "portal": "source-portal-name"
}

The system generates composite keys using {company}_{title} (lowercased with spaces converted to underscores) to uniquely identify postings regardless of URL variations. This guarantees that duplicate URLs or company-title combinations are ignored during subsequent scrape operations.

CSV Tracker Layer (Application History)

Permanent application records reside in job_search_tracker.csv at the repository root. As documented in [.claude/commands/outcome.md](.claude/commands/outcome.md), this file uses stable column ordering with fields including company, role, source, status, and date_applied. New columns are added additively to preserve backward compatibility, making this CSV the source of truth for the /html-report and /upskill commands.

How the Storage Layers Interact

The framework orchestrates data flow through three distinct operations:

1. Ingestion: Add postings by dropping text files into documents/postings/ or submitting URLs via the /apply command documented in [.claude/commands/apply.md](.claude/commands/apply.md)

2. Indexing: When /scrape runs, it reads portal CLI outputs, fetches full postings via WebFetch when needed, and writes structured JSON entries to job_scraper/seen_jobs.json

3. Application: Upon applying, the framework copies posting text to documents/applications/<company>_<role>/job_posting.md while appending metadata to job_search_tracker.csv

Agents preserve unknown keys when rewriting seen_jobs.json, allowing schema evolution without migration scripts.

Programmatically Working with the Schema

To manually add entries to the JSON index:

import json
from pathlib import Path
import datetime

def add_seen_job(url: str, title: str, company: str, portal: str, fit: str):
    path = Path("job_scraper/seen_jobs.json")
    data = {"seen": {}} if not path.exists() else json.loads(path.read_text())
    
    key = f"{company}_{title}".replace(" ", "_").lower()
    data["seen"][key] = {
        "title": title,
        "company": company,
        "url": url,
        "first_seen": datetime.date.today().isoformat(),
        "fit": fit,
        "status": "new",
        "portal": portal,
    }
    path.write_text(json.dumps(data, indent=2))

# Example usage

add_seen_job(
    url="https://jobindex.dk/job/1234567",
    title="AI Engineer",
    company="Acme Corp",
    portal="jobindex-search",
    fit="high",
)

For manual workflow integration:

# Create raw posting storage

mkdir -p documents/postings
cat <<EOF > documents/postings/Acme\ Corp\ -\ AI\ Engineer.txt
[Full job posting content here]
EOF

# Trigger indexing

/scrape

Summary

• The data schema for storing job postings spans three layers: unstructured text in documents/postings/, structured JSON in job_scraper/seen_jobs.json, and tabular history in job_search_tracker.csv
• The JSON schema enforces required fields for title, company, url, first_seen, fit (high/medium/low), status (new/skipped/evaluated/ranked/expired), and portal
• Composite keys prevent duplicates based on company-title combinations rather than URLs alone
• The CSV tracker maintains additive schema evolution, ensuring reporting commands continue functioning as new tracking fields are introduced
• Raw source text remains available in documents/ while agents work against lightweight JSON pointers

Frequently Asked Questions

What happens if I add custom fields to the JSON schema?

The framework handles schema evolution gracefully. When agents rewrite job_scraper/seen_jobs.json, they preserve unknown keys, allowing you to add fields like salary_range or remote_policy without breaking existing functionality or requiring migration scripts.

Can I store job postings without using the automated scraper?

Yes. Simply create text files in documents/postings/ following the <Company> - <Job Title>.txt naming convention. While these won't appear in the JSON index until processed by /scrape or /apply, the system treats them as valid raw postings for manual review.

How does the system prevent duplicate job postings?

The JSON index generates unique keys using the pattern {company}_{title} with spaces replaced by underscores and text normalized to lowercase. When the scraper encounters an existing company-title combination, it skips creating a new entry regardless of whether the URL differs from previous sightings.

Where is application history permanently stored?

Application history persists in job_search_tracker.csv at the repository root. This file serves as the authoritative source for the /outcome, /upskill, and /html-report commands, tracking every position you've applied to along with dates and outcomes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}