How to Add a New Job Board Provider to the Career-Ops Scanner

Create a new .mjs file in the providers/ directory that exports NAME, BASE_URL, an async search() function, and a parse() function conforming to the Career-Ops job schema, then run scan.mjs to automatically discover your provider.

Career-Ops is an open-source job aggregation scanner by santifer/career-ops that standardizes listings from multiple sources into a unified pipeline. To extend the scanner's reach beyond the built-in adapters, you can add a new job board provider by implementing a lightweight JavaScript module interface. The core scanner (scan.mjs) dynamically discovers these modules and orchestrates the search flow, making the integration process straightforward and modular.

Understanding the Provider Contract

Every provider module in the providers/ directory must implement a well-defined interface. The core scanner imports these modules and invokes their methods to fetch raw listings, normalize the data, and feed results into the deduplication pipeline.

Each provider must export the following members:

• NAME (string): Human-readable identifier displayed in logs and reports (e.g., "RemoteOK").
• BASE_URL (string): Root URL of the job board used for diagnostics and constructing absolute links.
• search(query, page = 1) (async function): Accepts a search string and optional page number, returns an array of raw job objects. Must handle HTTP errors and respect rate limits.
• parse(rawJob) (function): Transforms a raw job object into the canonical Career-Ops job schema containing at minimum {title, company, location, url, postedAt}.
• isActive(job) (optional function): Returns a boolean indicating whether the posting is still live; useful for filtering stale entries from boards that serve expired listings.

Providers must be pure ECMAScript modules (.mjs) with no side effects—no file system writes, no console.log statements, and no global state mutations—because the scanner executes them in parallel workers.

Implementing the New Job Board Provider

Step 1: Create the Provider Module

Create a file named providers/<your-board>.mjs using the following skeleton adapted from providers/remoteok.mjs:

// providers/exampleboard.mjs
export const NAME = "ExampleBoard";
export const BASE_URL = "https://exampleboard.com";

export async function search(query, page = 1) {
  const url = `${BASE_URL}/api/v1/jobs?search=${encodeURIComponent(query)}&page=${page}`;
  const resp = await fetch(url, { headers: { "Accept": "application/json" } });
  if (!resp.ok) throw new Error(`Failed to fetch ${url}: ${resp.status}`);
  const data = await resp.json();
  return data.jobs; // array of raw job objects
}

export function parse(raw) {
  return {
    title: raw.title,
    company: raw.company.name,
    location: raw.location || "Remote",
    url: raw.url,
    postedAt: new Date(raw.date_posted),
    // additional fields as needed by downstream modes
  };
}

// Optional: filter out closed or stale listings
export function isActive(job) {
  return !!job.applyUrl && !job.isClosed;
}

Step 2: Handle Authentication and Rate Limiting

If the job board requires an API key, read it from environment variables at runtime:

const token = process.env.EXAMPLE_BOARD_TOKEN;
const resp = await fetch(`${BASE_URL}/jobs`, {
  headers: { "Authorization": `Bearer ${token}` }
});

Add the variable name to .env.example in the repository root so users know to populate their local .env file. For rate-limited endpoints, implement a delay inside search() or utilize concurrency libraries like p-limit to throttle requests and avoid HTTP 429 errors.

Step 3: Normalize Data with parse()

The parse() function is critical for interoperability. It must return a plain JavaScript object matching the Career-Ops job schema with these standard fields:

• title: Job title string
• company: Employer name string
• location: Geographic location or "Remote"
• url: Direct link to the application or posting
• postedAt: JavaScript Date object representing publication time

Downstream modules like generate-pdf.mjs and analyze-patterns.mjs depend on this consistent structure to generate reports and detect duplicate postings across different providers.

Registering the Provider

The scanner discovers providers automatically via a glob pattern that imports every .mjs file in the providers/ directory. Simply placing your new module in this folder makes it available without modifying scan.mjs.

If you prefer explicit control over loading order or need to disable providers conditionally, edit the PROVIDERS array in scan.mjs:

import * as remoteok from "./providers/remoteok.mjs";
import * as myBoard from "./providers/exampleboard.mjs";

export const PROVIDERS = [remoteok, myBoard, /* other providers */];

This approach is useful when debugging a specific provider or managing dependencies between modules.

Testing Your Implementation

Create a unit test file providers/<your-board>.test.mjs to validate the contract before running the full scanner:

import assert from "node:assert";
import * as myBoard from "./exampleboard.mjs";

describe("ExampleBoard provider", () => {
  it("should return parsed jobs", async () => {
    const raw = await myBoard.search("engineer", 1);
    assert.ok(Array.isArray(raw), "search must return an array");
    
    const job = myBoard.parse(raw[0]);
    assert.ok(job.title, "parsed job must have a title");
    assert.ok(job.company, "parsed job must have a company");
    assert.ok(job.url, "parsed job must have a URL");
  });
});

Run the test suite using node test.mjs or the npm script defined in package.json to verify that your provider correctly handles API responses and produces valid job objects.

Running the Scanner

With your provider implemented and tested, execute the scanner to include your new job board in the aggregation pipeline:

node scan.mjs --query "backend engineer" --pages 3

The scanner will automatically invoke your provider's search() method, normalize results through parse(), and feed them into the deduplication and reporting flow. Listings from your new provider will appear alongside those from remoteok.mjs, lever.mjs, and other built-in adapters.

Summary

• Create a new file in providers/ with the .mjs extension to add a job board provider to Career-Ops.
• Export NAME, BASE_URL, search(), and parse() to satisfy the provider interface required by scan.mjs.
• Normalize raw API data into the canonical job schema in your parse() function to ensure compatibility with downstream report generators.
• Test your implementation with a dedicated test file before running the full scanner pipeline.
• Deploy by placing the file in the correct directory; the scanner discovers it automatically via glob pattern matching.

Frequently Asked Questions

What file extension should I use for the provider module?

Career-Ops uses ES modules, so you must save your provider as a .mjs file. The scanner specifically looks for files matching providers/*.mjs during the dynamic import phase. Standard .js files will not be discovered unless you explicitly import them into scan.mjs.

Do I need to modify scan.mjs to register my new provider?

No, the scanner automatically discovers any .mjs file placed in the providers/ directory through a glob pattern import loop. However, if you want to control the loading order or conditionally exclude certain providers, you can opt for explicit registration by importing the module and adding it to the PROVIDERS array in scan.mjs.

How do I handle job boards that require API authentication?

Store sensitive credentials in a .env file and access them via process.env.YOUR_PROVIDER_TOKEN inside your search() function. Make sure to add the variable name to .env.example in the repository root so other users know which environment variables are required. Never hardcode API keys directly into the provider module.

Why is my provider throwing errors when the scanner runs it in parallel?

Career-Ops executes providers in parallel workers, so your module must be free of side effects like console.log or file system writes. If you encounter rate limiting from the job board API, implement a delay using setTimeout or use a concurrency limiting library like p-limit to throttle requests within your search() implementation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}