How the Marin API Works: Architecture and Endpoints Explained

The Marin API provides a FastAPI-based HTTP interface that converts JSON payloads into internal dataclasses and orchestrates pipeline execution through a modular step runner.

The marin-community/marin repository implements a modular pipeline framework designed for data-processing and model-training workflows. The Marin API exposes these capabilities through a programmatic REST interface, allowing users to create experiments, monitor distributed sweep progress, and retrieve profiling data through standardized HTTP endpoints.

Core Architecture and Entry Points

The Marin API follows a layered architecture that separates HTTP transport from business logic. The web layer resides in the lib/marin/src/marin/web/ package and handles request routing, validation, and dispatch.

FastAPI Application Registration

The primary entry point lives in lib/marin/src/marin/web/__init__.py. This module instantiates the FastAPI application and mounts the route handlers that constitute the public API. When the server starts, this file registers endpoints such as POST /v1/experiments and GET /v1/sweeps/{sweep_id}.

Request Conversion and Validation

Incoming JSON undergoes type conversion in lib/marin/src/marin/web/convert.py. This utility module translates raw HTTP payloads into internal Marin objects—specifically Experiment, Sweep, and StepSpec instances. The conversion relies on Pydantic models defined in the lib/marin/src/marin/experiment module to enforce schema validation before objects pass to the execution engine.

API Request Lifecycle

When a client issues a request to the Marin API, the framework processes it through three distinct stages:

1. Routing: The FastAPI router receives the request and validates the HTTP method and path against handlers registered in lib/marin/src/marin/web/__init__.py.
2. Deserialization: The convert.py module transforms validated JSON into strongly-typed dataclasses using the Pydantic models.
3. Execution: The request dispatches to the engine in lib/marin/src/marin/execution/step_runner.py, which orchestrates the actual work and optionally delegates to external backends such as Iris or VLLM.

CRUD Operations and Endpoint Examples

The Marin API supports full CRUD operations for experiments, sweeps, checkpoints, and profiling data. The following examples demonstrate authenticated requests using Python's requests library.

Creating an Experiment

Submit a new experiment configuration via POST /v1/experiments:

import requests

exp_payload = {
    "name": "my_experiment",
    "config": {"model": "gpt-2", "batch_size": 32},
}
resp = requests.post(
    "https://api.marin.dev/v1/experiments",
    json=exp_payload,
    headers={"Authorization": "Bearer <TOKEN>"},
)
print(resp.json())   # => {"id": "...", "status": "created"}

Monitoring Sweep Status

Query the execution state of a running sweep using GET /v1/sweeps/{sweep_id}:

sweep_id = "sweep-123"
status = requests.get(
    f"https://api.marin.dev/v1/sweeps/{sweep_id}",
    headers={"Authorization": "Bearer <TOKEN>"},
).json()
print(status)        # => {"state": "RUNNING", "progress": 0.57}

Uploading Profiling Data

Submit performance metrics via POST /v1/profiling:

profile = {"step": "train", "latency_ms": 123, "memory_mb": 2048}
requests.post(
    "https://api.marin.dev/v1/profiling",
    json=profile,
    headers={"Authorization": "Bearer <TOKEN>"},
)

Response Serialization

The Marin API returns JSON-serialized responses using a custom encoder defined in lib/marin/src/marin/utilities/json_encoder.py. This encoder handles complex datatypes that standard JSON libraries cannot process, ensuring consistent serialization of experiment states, sweep configurations, and checkpoint metadata.

Summary

• Entry Point: lib/marin/src/marin/web/__init__.py registers the FastAPI application and public routes.
• Type Conversion: lib/marin/src/marin/web/convert.py transforms JSON payloads into validated Experiment, Sweep, and StepSpec objects.
• Execution Engine: lib/marin/src/marin/execution/step_runner.py orchestrates pipeline steps and delegates to external backends when required.
• Serialization: lib/marin/src/marin/utilities/json_encoder.py provides custom JSON encoding for API responses.
• Endpoint Coverage: The API exposes CRUD operations for experiments, sweeps, checkpoints, and profiling data through standard HTTP methods.

Frequently Asked Questions

What authentication method does the Marin API require?

The Marin API implements Bearer token authentication. Clients must include an Authorization header with the value Bearer <TOKEN> in every HTTP request to authenticate against the endpoints.

Which file handles the conversion from JSON to internal Marin objects?

The lib/marin/src/marin/web/convert.py module contains the utility functions that translate incoming JSON payloads into internal dataclasses after they pass Pydantic validation in the lib/marin/src/marin/experiment module.

Can the Marin API delegate execution to external backends?

Yes. According to the source code in lib/marin/src/marin/execution/step_runner.py, the execution engine orchestrates workloads and can optionally delegate processing to external backends such as Iris or VLLM for specialized computation.

What content type does the Marin API return?

The Marin API returns JSON-serialized responses using the custom encoder located in lib/marin/src/marin/utilities/json_encoder.py, which handles complex datatypes beyond standard JSON serialization capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}