# marin | The Marin Project | Knowledge Base | Instagit

Open-source framework for the research and development of foundation models.

GitHub Stars: 2.5k

Repository: https://github.com/marin-community/marin

---

## Articles

### [How to Perform Text Deduplication with Marin-dupekit: 3 Methods Explained](/marin-community/marin/how-to-perform-text-deduplication-with-marin-dupekit)

Learn to perform text deduplication with Marin-dupekit. Discover three distributed modes Exact Paragraph Exact Document and Fuzzy Document for petabyte-scale corpora using the Zephyr execution engine.

- Tags: how-to-guide
- Published: 2026-09-10

### [What Is the Marin‑Ducky Service? A DuckDB‑Powered SQL Dashboard for Cloud Storage](/marin-community/marin/what-is-the-marin-ducky-service-for)

Explore the Marin-Ducky service a DuckDB-powered SQL dashboard for querying data in Google Cloud Storage and other object stores. Run ad-hoc SQL queries easily.

- Tags: getting-started
- Published: 2026-09-10

### [How to Deploy Models with Marin-Deploy: A Complete Guide to ML Model Serving](/marin-community/marin/how-to-deploy-models-with-marin-deploy)

Deploy ML models effortlessly with Marin-Deploy. This guide shows how to use its command-line interface for seamless deployment to Kubernetes and cloud platforms.

- Tags: how-to-guide
- Published: 2026-09-10

### [What Is the Purpose of Marin-core? A Deep Dive into the Foundation of the Marin Ecosystem](/marin-community/marin/what-is-the-purpose-of-marin-core)

Discover the purpose of Marin-core, the foundational data layer for the Marin ecosystem. It standardizes data for question-answer examples and conversation logs, ensuring smooth data flow across pipelines.

- Tags: deep-dive
- Published: 2026-09-10

### [How Marin Ensures Training Mixture Reproducibility: A Two‑Phase Pipeline](/marin-community/marin/how-does-marin-ensure-training-mixture-reproducibility)

Marin ensures training mixture reproducibility with a two phase pipeline. Discover how it guarantees identical logical mixtures for consistent results.

- Tags: internals
- Published: 2026-09-10

### [Where to Find Delphi Checkpoints on Hugging Face: A Complete Guide](/marin-community/marin/where-can-i-find-delphi-checkpoints-on-hugging-face)

Easily find Delphi checkpoints on Hugging Face. Discover specific model identifiers in the Marin repository for your NLP projects.

- Tags: how-to-guide
- Published: 2026-09-10

### [Delphi: Marin’s Open Scaling Suite for Systematic LLM Training](/marin-community/marin/what-is-delphi-marins-open-scaling-suite)

Discover Delphi, Marin's open scaling suite for systematic LLM training. Automate reproducible pipelines from 3×10¹⁸ to 10²³ FLOPs with a learned scaling law.

- Tags: deep-dive
- Published: 2026-09-10

### [How Marin Handles Text Deduplication: Exact, Paragraph, and Fuzzy Matching Explained](/marin-community/marin/how-does-marin-handle-text-deduplication)

Discover how Marin handles text deduplication with exact paragraph, exact document, and fuzzy matching. Learn about its powerful Zephyr engine and MinHash algorithms.

- Tags: deep-dive
- Published: 2026-09-10

### [How Marin Curates Raw Source Data](/marin-community/marin/how-is-raw-source-data-curated-in-marin)

Learn how Marin curates raw source data through a deterministic download transform normalize pipeline. Ensure reproducibility and traceability with Datakit and StepSpec.

- Tags: how-to-guide
- Published: 2026-09-10

### [Haliax for Named Array Programming in JAX: A Complete Guide](/marin-community/marin/what-is-haliax-for-named-array-programming-in-jax)

Discover Haliax for named array programming in JAX. Replace positional dimensions with named axes to eliminate shape mismatch bugs and write self-documenting code.

- Tags: deep-dive
- Published: 2026-09-10

### [How Marin-Fray Provides a Distributed Execution Substrate](/marin-community/marin/how-does-marin-fray-provide-distributed-execution-substrate)

Marin-Fray provides a distributed execution substrate by unifying Iris job orchestration. Learn how it translates specifications, supports actor hosting, coscheduling, and dynamic discovery.

- Tags: architecture
- Published: 2026-09-10

### [Marin-Zephyr Pipeline Stages: A Complete Guide to the 8 Core Stage Types](/marin-community/marin/what-stages-are-involved-in-the-marin-zephyr-pipeline)

Explore the 8 core Marin-Zephyr pipeline stages: MAP, FILTER, FLATMAP, REDUCE, GROUP_BY, SORTED_MERGE_JOIN, WRITE, and CUSTOM. Understand how ZephyrCoordinator orchestrates these steps.

- Tags: how-to-guide
- Published: 2026-09-10

### [How to Use Marin-Zephyr for Dataset Processing: A Complete Guide](/marin-community/marin/how-to-use-marin-zephyr-for-dataset-processing)

Learn how to use Marin Zephyr for efficient dataset processing. This guide covers building scalable ETL pipelines with a high-level API. No worker orchestration needed.

- Tags: how-to-guide
- Published: 2026-09-10

### [How Marin-Iris Manages Distributed Job Execution: Architecture and Implementation](/marin-community/marin/how-does-marin-iris-manage-distributed-job-execution)

Learn how Marin Iris manages distributed job execution across local and cloud GPUs and Kubernetes. Discover its controller-database-scheduler architecture for consistent state.

- Tags: architecture
- Published: 2026-09-10

### [What Are the Custom CUDA/TPU Kernels in Levanter? A Deep Dive into the Kernel Library](/marin-community/marin/what-are-the-custom-cuda-tpu-kernels-in-levanter)

Explore Levanter's custom CUDA and TPU kernels for GPUs and TPUs. Discover high-performance primitives like DeepPE transport, fused cross-entropy, and SSD attention for accelerated deep learning.

- Tags: deep-dive
- Published: 2026-09-10

### [What Are the Key Libraries in Marin? Inside the Python Workspace Architecture](/marin-community/marin/what-are-the-key-libraries-in-marin)

Explore the key Python libraries in Marin, including marin-iris, marin-levanter, and marin-haliax. Understand the Marin workspace architecture and its specialized components for efficient LLM training and cluster orchestration.

- Tags: deep-dive
- Published: 2026-09-10

### [How Marin's StepRunner Handles DAG Execution: A Deep Dive into the Core Scheduler](/marin-community/marin/how-does-marin-steprunner-handle-dag-execution)

Discover how Marin's StepRunner handles DAG execution. Learn about its eager pulling, dependency expansion, and parallel execution with thread pools and caching.

- Tags: deep-dive
- Published: 2026-09-10

### [Marin End-to-End Pipeline: Core Components and Architecture Guide](/marin-community/marin/what-are-the-components-of-the-marin-end-to-end-pipeline)

Explore the Marin end-to-end pipeline, detailing core components like Datakit, Zephyr, and Iris for data ingestion, distributed training, and model evaluation.

- Tags: architecture
- Published: 2026-09-10

### [How to Build Foundation Models with Marin: A Complete Guide to the Open-Source Training Stack](/marin-community/marin/how-to-build-foundation-models-with-marin)

Build foundation models at any scale with Marin, the open-source JAX-based distributed training framework. Learn how to create reproducible models with this complete guide.

- Tags: how-to-guide
- Published: 2026-09-10

### [What Is the Marin Research Platform? A Technical Guide to Open-Source Foundation Model Development](/marin-community/marin/what-is-marin-research-platform)

Explore the Marin research platform, an open-source framework for reproducible foundation model development. Unify data curation, pre-training, and evaluation in streamlined pipelines.

- Tags: technical-guide
- Published: 2026-09-10

### [How the Delphi Scaling Suite Maps Compute Budgets to Model Configurations](/marin-community/marin/delphi-scaling-suite-compute-budgets-model-configurations)

Discover how the Delphi scaling suite maps compute budgets to model configurations using a power-law scaling recipe fitted empirical relationships in the marin-community/marin repository.

- Tags: architecture
- Published: 2026-08-29

### [How MoE Expert Load Balancing Works with Quantile Balancing in Marin](/marin-community/marin/moe-expert-load-balancing-quantile-balancing)

Explore how MoE expert load balancing leverages quantile balancing in Marin for even token distribution. Learn about dynamic threshold setting with TOPK or HIST estimators.

- Tags: deep-dive
- Published: 2026-08-29

### [How to Query Finelog for CPU and Memory Profile Captures](/marin-community/marin/query-finelog-cpu-memory-profile-captures)

Learn to query Finelog for CPU and memory profile captures using SQL on iris.task and iris.profile tables. Retrieve memory_peak_mb and cpu_time_sec metrics easily.

- Tags: how-to-guide
- Published: 2026-08-29

### [How Zephyr Dataset Normalization Handles Pipeline Boundaries](/marin-community/marin/zephyr-dataset-normalization-pipeline-boundaries)

Discover how Zephyr dataset normalization handles pipeline boundaries by normalizing paths for deterministic, platform-independent matching between readers and writers.

- Tags: internals
- Published: 2026-08-29

### [WORKER vs KUBERNETES Backend Kinds in Iris: Execution Models Explained](/marin-community/marin/worker-vs-kubernetes-backend-kind-differences-iris)

Explore Iris WORKER vs KUBERNETES backend kinds. Understand their distinct execution models for managing workloads and discover which fits your needs.

- Tags: deep-dive
- Published: 2026-08-29

### [How Iris Autoscaler Worker Provisioning and Scaling Works: A Code-Level Breakdown](/marin-community/marin/iris-autoscaler-worker-provisioning-scaling-logic)

Explore the Iris autoscaler's code-level logic for worker provisioning and scaling. Understand how it manages cluster state, job demand, capacity, and failure handling.

- Tags: internals
- Published: 2026-08-29

### [SQLite Schema Migration Pattern in the Iris Controller: Versioned Upgrades Explained](/marin-community/marin/sqlite-schema-migration-pattern-iris-controller)

Explore the SQLite schema migration pattern in the Iris controller. Learn about versioned upgrades, meta tables, and sequential Python script execution for safe database updates.

- Tags: how-to-guide
- Published: 2026-08-29

### [How to Add a New Cluster Backend Provider to Iris: Complete Implementation Guide](/marin-community/marin/add-new-cluster-backend-provider-iris)

Learn how to add a new cluster backend provider to Iris. This guide covers implementing TaskBackend, registering your provider, and updating the CLI for seamless integration. Enhance Iris functionality today.

- Tags: how-to-guide
- Published: 2026-08-29

### [How Iris Federation Routing Works Across Cross-Cluster Storage](/marin-community/marin/iris-federation-routing-cross-cluster-storage)

Learn how Iris federation routing classifies jobs as LOCAL QUEUE or REJECT using PeerRouter and cluster pins. Optimize cross-cluster storage efficiency.

- Tags: deep-dive
- Published: 2026-08-29

### [Finelog Stats Namespaces for Task and Worker Telemetry in Marin: A Complete Reference](/marin-community/marin/finelog-stats-namespaces-task-worker-telemetry)

Discover Finelog stats namespaces for task and worker telemetry in Marin. Learn how zephyr.stage and zephyr.worker capture essential metrics for distributed Zephyr executions.

- Tags: api-reference
- Published: 2026-08-29

### [How to Deploy the Iris Controller to CoreWeave Kubernetes: A Complete Guide](/marin-community/marin/deploy-iris-controller-coreweave-kubernetes)

Deploy the Iris controller to CoreWeave Kubernetes. Follow this guide to configure your cluster YAML, build the controller image, and apply manifests for seamless deployment.

- Tags: how-to-guide
- Published: 2026-08-29

### [How Marin Implements Disk Caching and Artifact Fingerprinting](/marin-community/marin/marin-disk-caching-artifact-fingerprinting)

Discover how Marin implements disk caching and artifact fingerprinting using distinct systems for cloud storage persistence and deterministic hashing, ensuring efficient artifact management.

- Tags: internals
- Published: 2026-08-29

### [How to Configure AdamConfig and Optimizer Settings in Levanter](/marin-community/marin/configure-adamconfig-optimizer-settings-levanter)

Learn to configure AdamConfig and optimizer settings in Levanter for learning rate schedules, weight decay, gradient clipping, and step skipping. Build Optax optimizers easily.

- Tags: how-to-guide
- Published: 2026-08-29

### [How to Write and Run Tests Following Marin's Testing Policy: A Complete Guide](/marin-community/marin/write-run-tests-marin-testing-policy)

Learn how to write and run tests following Marin's testing policy. Master behavior-focused integration tests and ensure reliable validation with `uv run pytest`.

- Tags: how-to-guide
- Published: 2026-08-29

### [How Iris Handles Scheduling Policy and Per‑User Budget Enforcement](/marin-community/marin/iris-scheduling-policy-per-user-budget-enforcement)

Iris enforces scheduling policy and per-user budget enforcement via a three-stage pipeline. Learn how it dynamically downgrades priority and round-robins tasks to prevent monopolization.

- Tags: how-to-guide
- Published: 2026-08-29

### [Dependency Hierarchy Between Iris, Levanter, Zephyr, and Marin: Runtime Stack Explained](/marin-community/marin/dependency-hierarchy-iris-levanter-zephyr-marin)

Understand the dependency hierarchy of Iris, Levanter, Zephyr, and Marin. Explore how these libraries form the marin-community/marin runtime stack for ML pipelines.

- Tags: internals
- Published: 2026-08-29

### [How Iris Environment Variables Are Injected Into Docker Containers](/marin-community/marin/iris-environment-variable-injection-docker-containers)

Learn how Iris injects environment variables into Docker containers. Discover the two-phase pipeline for build-time metadata and run-time configuration.

- Tags: how-to-guide
- Published: 2026-08-29

### [Understanding the Lazy Evaluation Pattern in Marin: How `lower()` Works](/marin-community/marin/lazy-evaluation-pattern-lower-marin-execution-pipeline)

Discover the lazy evaluation pattern in Marin and how lower() transforms pipeline handles into execution specs for inspection, caching, and distributed planning before ML training.

- Tags: internals
- Published: 2026-08-29

### [How to Troubleshoot Iris Cluster Issues Using OPS.md Diagnostic Patterns](/marin-community/marin/troubleshoot-iris-cluster-ops-md-diagnostic-patterns)

Troubleshoot Iris cluster issues with OPS.md diagnostic patterns. Explore CLI commands and RPC endpoints for deep system inspection, from scheduler state to process logs. Learn more now.

- Tags: how-to-guide
- Published: 2026-08-29

### [What Is Fray ResourceConfig and How Does It Enable Distributed Execution?](/marin-community/marin/fray-resourceconfig-distributed-execution)

Learn how Fray ResourceConfig enables distributed execution by declaratively defining hardware requirements for tasks in marin-community/marin jobs.

- Tags: internals
- Published: 2026-08-29

### [How to Debug Zephyr Pipeline Stragglers and Coordinator Issues in marin-community/marin](/marin-community/marin/debug-zephyr-pipeline-stragglers-coordinator-issues)

Debug Zephyr pipeline stragglers and coordinator issues in marin-community/marin. Monitor counters and heartbeat loss to identify and resolve problems quickly.

- Tags: how-to-guide
- Published: 2026-08-29

### [How to Add a New Dataset to Marin Using Agent Skills](/marin-community/marin/add-new-dataset-marin-agent-skills)

Learn to add a new Hugging Face dataset to Marin using the add dataset agent skill. Inspect schema, create lazy-builder modules, and validate registration easily.

- Tags: how-to-guide
- Published: 2026-08-29

### [How StepRunner Executes Experiment Steps in Topological Order in Marin](/marin-community/marin/how-steprunner-executes-experiment-steps-in-topological-order)

Discover how StepRunner executes experiment steps in topological order within Marin. Learn about DAGs, Kahn's algorithm, and efficient dependency management for seamless experiment runs.

- Tags: internals
- Published: 2026-08-29

### [Marin Development Roadmap: LoRA, MoE, and Infrastructure Automation Plans](/marin-community/marin/marin-development-roadmap)

Explore the Marin development roadmap focusing on LoRA, MoE, and infrastructure automation for efficient model fine-tuning and cost management. See active implementation details.

- Tags: roadmap
- Published: 2026-08-29

### [How to Get Support for Marin: Community Channels and Resources](/marin-community/marin/how-to-get-support-marin)

Get expert support for Marin via Discord, GitHub Issues, and comprehensive ReadTheDocs. Find quick solutions and contribute to the project easily. Access community channels now.

- Tags: getting-started
- Published: 2026-08-29

### [Security Implications of Using Marin: Token Validation and Container Isolation Explained](/marin-community/marin/marin-security-implications)

Explore Marin's security features: learn how JWTs, audience validation, and container isolation prevent authentication forgery, privilege escalation, and container breakouts.

- Tags: security-implications
- Published: 2026-08-29

### [How to Integrate Marin with Other Tools: A Complete Guide to Modular Pipeline Extensions](/marin-community/marin/how-to-integrate-marin-with-other-tools)

Learn how to integrate Marin with other tools by creating wrapper classes and wiring them into your pipeline. This guide covers modular pipeline extensions for seamless integration.

- Tags: how-to-guide
- Published: 2026-08-29

### [Performance Considerations for Marin: Scaling Laws, Architecture, and Hardware Tuning](/marin-community/marin/marin-performance-considerations)

Explore Marin's performance considerations including scaling laws, architecture, and hardware tuning. Learn how Marin optimizes training throughput through efficient service separation, data caching, and storage selection.

- Tags: performance
- Published: 2026-08-29

### [Best Practices for Developing with Marin: A Complete Guide to LLM Pipelines](/marin-community/marin/best-practices-developing-with-marin)

Master Marin development with our guide to LLM pipelines. Learn best practices for architecture, configurations, and dependencies to build robust applications.

- Tags: best-practices
- Published: 2026-08-29

### [How to Update Marin to the Latest Version: A Step-by-Step Guide for the Monorepo](/marin-community/marin/how-to-update-marin-latest-version)

Update Marin to the latest version with this step-by-step guide. Learn how to pull new commits and refresh workspace dependencies using uv sync for your monorepo.

- Tags: how-to-guide
- Published: 2026-08-29

### [Example Projects Using Marin: 4 Ready-to-Run Tutorials](/marin-community/marin/example-projects-using-marin)

Explore example projects using Marin with four ready-to-run tutorials. Learn lazy artifact pipelines, tokenization, and model training with StepRunner and ArtifactStep.

- Tags: tutorial
- Published: 2026-08-29

### [Marin Release Cycle: Daily Development Builds, Stable Releases, and Canary Testing](/marin-community/marin/marin-release-cycle)

Discover the Marin release cycle: daily development builds, stable releases via Git tags, and canary tests for validated PyPI publication.

- Tags: release-cycle
- Published: 2026-08-29

### [How to Debug Issues in Marin: A Complete Troubleshooting Guide](/marin-community/marin/how-to-debug-issues-marin)

Troubleshoot Marin issues effectively. Learn to debug with Iris CLI, analyze flamegraphs, and restore from controller checkpoints. Master Marin troubleshooting today.

- Tags: how-to-guide
- Published: 2026-08-29

### [Marin Configuration Options: A Complete Guide to `config/marin.yaml`](/marin-community/marin/marin-configuration-options)

Explore marin configuration options in config marin yaml. Discover how to set Iris cluster name, GCS bucket mappings, and temporary file TTL for optimal runtime behavior.

- Tags: how-to-guide
- Published: 2026-08-29

### [How to Use the Marin CLI: Command Reference and Workflow Guide](/marin-community/marin/how-to-use-marin-cli)

Master the Marin CLI with this guide to its commands and workflows. Manage training, profiling, validation, and inference for your ML projects efficiently.

- Tags: how-to-guide
- Published: 2026-08-29

### [Main Modules and Packages in the Marin Repository: A Complete Architecture Guide](/marin-community/marin/marin-main-modules-packages)

Explore the main modules and packages in the Marin repository. Discover how these nine Python libraries form a layered stack for large-scale language model research from data ingestion to distributed training.

- Tags: architecture
- Published: 2026-08-29

### [Where to Find Marin Project Documentation: Complete Guide to the Repository docs/ Directory](/marin-community/marin/where-to-find-marin-project-documentation)

Find Marin project documentation easily within the repository's docs/ directory. This guide shows you the entry point and online hosting location for all Marin docs.

- Tags: getting-started
- Published: 2026-08-29

### [How to Install Marin on Linux, macOS, and Windows: The Complete Guide](/marin-community/marin/how-to-install-marin-different-os)

Install Marin on Linux macOS and Windows with this comprehensive guide. Learn how to set up Marin using uv for environment management and ensure smooth installation across all platforms.

- Tags: getting-started
- Published: 2026-08-29

### [What Is the License for the Marin Project? Complete Guide to Apache 2.0](/marin-community/marin/marin-project-license)

Understand the Marin project license. Explore the Apache 2.0 license, its broad usage rights, and patent protection in this complete guide.

- Tags: api-reference
- Published: 2026-08-29

### [How to Contribute to the Marin Project: A Complete Developer’s Guide](/marin-community/marin/how-to-contribute-marin-project)

Learn how to contribute to the Marin project by cloning the repo, setting up Python 3.12, installing dependencies, and following our detailed guide to open a pull request. Start contributing today!

- Tags: how-to-guide
- Published: 2026-08-29

### [What Programming Languages Are Used in Marin? A Multi-Language Codebase Breakdown](/marin-community/marin/programming-languages-used-in-marin)

Explore the Marin codebase and discover the programming languages powering its architecture: Python, Rust, JavaScript, and Shell. Understand the role of each language in this multi-language project.

- Tags: codebase-breakdown
- Published: 2026-08-29

### [How to Run Tests in the Marin Project: A Complete Guide](/marin-community/marin/how-to-run-tests-marin-project)

Learn how to run tests in the Marin project. Use `uv run pytest` for single files or `uv run --no-project infra/ci/run_tests.py` for the complete safe test suite. Ensure your code works flawlessly.

- Tags: how-to-guide
- Published: 2026-08-29

### [Marin Repository Directory Structure: A Complete Guide to the Multi-Layered Python Codebase](/marin-community/marin/marin-repository-directory-structure)

Explore the marin repository directory structure. Understand the layered Python codebase organizing core libraries, infrastructure, experiments, and docs for efficient machine learning development.

- Tags: deep-dive
- Published: 2026-08-29

### [Core Dependencies for Marin: Understanding the Multi-Package Python Workspace](/marin-community/marin/marin-core-dependencies)

Discover the eleven core dependencies for Marin, the multi-package Python workspace that powers your machine learning execution engine. Learn about key packages like iris, levanter, and ducky.

- Tags: deep-dive
- Published: 2026-08-29

### [How to Set Up a Local Development Environment for Marin](/marin-community/marin/how-to-set-up-local-dev-environment-marin)

Set up your local development environment for Marin quickly. Learn to install Python, uv, Git, clone the repo, and configure dependencies for seamless Marin development.

- Tags: getting-started
- Published: 2026-08-29

### [How the Marin Experiment Data Module Handles Tokenized Dataset Caching](/marin-community/marin/how-does-the-marin-experiment-data-module-handle-tokenized-dataset-caching)

Discover how the marin experiment data module uses lazy ArtifactStep handles and cryptographic fingerprinting for automated tokenized dataset caching, allowing efficient cache reuse.

- Tags: internals
- Published: 2026-08-28

### [How Marin Manages Tokenizer Configuration with MarinTokenizer](/marin-community/marin/how-does-marin-manage-tokenizer-configuration-with-marin-tokenizer)

Discover how Marin manages tokenizer configuration using MarinTokenizer. Learn to load Hugging Face tokenizers, add custom tokens, and apply chat templates efficiently.

- Tags: how-to-guide
- Published: 2026-08-28

### [Pulumi Infrastructure Pattern for Deploying Marin Clusters: Complete Implementation Guide](/marin-community/marin/what-is-the-pulumi-infrastructure-pattern-for-deploying-marin-clusters)

Learn the Pulumi infrastructure pattern for deploying Marin clusters. Separate resources into stacks, centralize shared GCP resources, and use GCS state backend with GCP KMS encryption for secure and efficient deployments.

- Tags: how-to-guide
- Published: 2026-08-28

### [How Zephyr Detects and Normalizes OOM Conditions in Stack Traces for Marin](/marin-community/marin/how-does-zephyr-normalize-and-detect-oom-conditions-in-stack-traces-for-marin)

Learn how Zephyr detects and normalizes OOM conditions in stack traces for Marin. Scrubbing memory addresses and file paths with regex ensures accurate error reporting.

- Tags: how-to-guide
- Published: 2026-08-28

### [How Marin Configures and Runs lm-evaluation-harness Evaluations](/marin-community/marin/how-does-marin-configure-and-run-lm-evaluation-harness-evals)

Learn how Marin configures and runs lm-evaluation-harness evaluations using a uv environment and CLI arguments for OpenAI-compatible endpoints.

- Tags: how-to-guide
- Published: 2026-08-28

### [How Finestore Manages Long-Term Artifact Storage and Lazy Artifact Retrieval in Marin](/marin-community/marin/how-does-finestore-manage-long-term-artifact-storage-and-lazy-artifact-retrieval-in-marin)

Discover how Finestore ensures durable artifact storage and lazy retrieval in Marin using immutable Parquet archives and efficient ReadViews. Learn about its advanced persistence and I/O deferral.

- Tags: deep-dive
- Published: 2026-08-28

### [How Rigging Provides Health Checks and Server Runtime Primitives in Marin](/marin-community/marin/how-does-rigging-provide-health-checks-and-server-runtime-primitives-in-marin)

Discover how Rigging enhances Marin services with health checks and server runtime primitives like rate limiting and secure tunneling. Improve your application's reliability and performance today.

- Tags: how-to-guide
- Published: 2026-08-28

### [How dupekit Performs Deduplication on Curated Datasets in Marin: A Technical Deep Dive](/marin-community/marin/how-does-dupekit-perform-deduplication-on-curated-datasets-for-marin)

Discover how dupekit performs deduplication on curated datasets in Marin using a hybrid approach of fuzzy MinHash-LSH and exact xxh3 hashing for efficient data cleaning.

- Tags: deep-dive
- Published: 2026-08-28

### [How Marin Handles Training Loss Spikes and Health Alerts](/marin-community/marin/how-does-marin-handle-training-loss-spikes-and-health-alerts)

Marin detects training loss spikes by comparing recent windows against baselines and triggers health alerts when loss floors exceed defined thresholds. Learn how Marin monitors your training.

- Tags: how-to-guide
- Published: 2026-08-28

### [Marin Integration Test: Running a Full LM Pipeline in Under 10 Minutes](/marin-community/marin/how-does-the-marin-integration-test-exercise-the-full-lm-pipeline-in-under-10-minutes)

Learn how the Marin integration test runs a full LM pipeline in under 10 minutes. Discover its efficient orchestration of cached steps and minimal GPT-2 setup for rapid testing.

- Tags: how-to-guide
- Published: 2026-08-28

### [Delphi Scaling Law Methodology in Marin: Predicting Large Model Performance with IsoFLOP Analysis](/marin-community/marin/what-is-the-delphi-scaling-law-methodology-for-predicting-large-model-performance-in-marin)

Discover the Delphi scaling law methodology in Marin for predicting large model performance. Learn how IsoFLOP analysis forecasts training configurations for any FLOP budget.

- Tags: deep-dive
- Published: 2026-08-28

### [How Marin Implements Mixture-of-Experts Quantile Balancing: A Technical Deep Dive](/marin-community/marin/how-does-marin-implement-mixture-of-experts-quantile-balancing)

Discover how Marin implements mixture-of-experts quantile balancing with QB routers, dynamic thresholds, and expert load estimation to prevent overload. A technical deep dive.

- Tags: deep-dive
- Published: 2026-08-28

### [How the Marin Service Architecture Isolates Clients from Transport Details](/marin-community/marin/how-does-the-marin-service-architecture-isolate-clients-from-transport-details)

Discover how the Marin service architecture uses a pluggable layer to isolate clients from transport details like HTTP or gRPC. Inject transports easily without client code changes.

- Tags: architecture
- Published: 2026-08-28

### [How Finelog's LogClient Writes Batches and Queries Structured Logs in Marin](/marin-community/marin/how-does-finelogs-logclient-write-batches-and-query-structured-logs-in-marin)

Learn how Finelog's LogClient writes batches and queries structured logs in Marin. Discover its Python facade for Arrow batch conversion, asynchronous writing, and efficient log retrieval via SQL or RPC.

- Tags: how-to-guide
- Published: 2026-08-28

### [How Zephyr Processes and Normalizes Raw Datasets for Training in Marin](/marin-community/marin/how-does-zephyr-process-and-normalize-raw-datasets-for-training-in-marin)

Learn how Zephyr processes and normalizes raw datasets for Marin training using a declarative API. Explore glob expansion, file specification, and format readers for efficient data pipelines.

- Tags: how-to-guide
- Published: 2026-08-28

### [Understanding Haliax's Role as the Core Tensor/Named Axis Library in Marin](/marin-community/marin/what-is-haliaxs-role-as-the-core-tensor-named-axis-library-in-marin)

Explore Haliax, Marin's core tensor and named axis library. Learn how it simplifies tensor operations with explicit named axes for clearer, more robust code.

- Tags: deep-dive
- Published: 2026-08-28

### [How Levanter Compiles and Scales JAX Training Across TPU Pods in Marin](/marin-community/marin/how-does-levanter-compile-and-scale-jax-training-across-tpu-pods-in-marin)

Discover how Levanter compiles and scales JAX training across TPU pods in Marin. It automatically detects topology, builds device meshes, and compiles SPMD XLA executables for efficient distributed training.

- Tags: internals
- Published: 2026-08-28

### [How Fray's ResourceConfig Allocates Distributed Compute Resources for Marin](/marin-community/marin/how-does-frays-resourceconfig-allocate-distributed-compute-resources-for-marin)

Learn how Fray's ResourceConfig efficiently allocates distributed compute resources for Marin by converting task requirements into Iris scheduler protobufs for cluster replica distribution.

- Tags: how-to-guide
- Published: 2026-08-28

### [How the iris:// and gcp:// URL Schemes Enable Service Addressing in Marin](/marin-community/marin/how-do-the-iris-and-gcp-url-schemes-work-for-service-addressing-in-marin)

Learn how iris:// and gcp:// URL schemes in Marin abstract service locations for cluster-aware resolution and backward compatibility with host:port addresses.

- Tags: internals
- Published: 2026-08-28

### [How Marin EndpointsService Enables Service Discovery and Lazy Resolution](/marin-community/marin/how-does-the-endpointsservice-enable-service-discovery-and-lazy-resolution-in-marin)

Learn how Marin's EndpointsService offers service discovery and lazy resolution. It resolves abstract URIs to HTTP URLs on demand, reducing network overhead during configuration parsing.

- Tags: internals
- Published: 2026-08-28

### [How Iris Handles Multi-Cluster Job Scheduling Across GCP and CoreWeave](/marin-community/marin/how-does-iris-handle-multi-cluster-job-scheduling-across-gcp-and-coreweave)

Learn how Iris achieves multi cluster job scheduling across GCP and CoreWeave using a three-stage pipeline classification placement and handoff for efficient resource allocation.

- Tags: how-to-guide
- Published: 2026-08-28

### [Understanding Marin’s Lazy Execution Model and When to Use `lower()`](/marin-community/marin/what-is-the-lazy-execution-model-in-marin-execution-lazy-and-when-should-i-use-lower)

Explore Marin's lazy execution model and learn when to use lower() for custom orchestration debugging and integration with external runners. Materialize StepSpecs explicitly.

- Tags: internals
- Published: 2026-08-28

### [How Marin StepRunner Executes Experiment Steps in Topological Order](/marin-community/marin/how-does-marins-steerunner-execute-experiment-steps-in-topological-order)

Discover how Marin StepRunner executes experiment steps in topological order by building a DAG, sorting topologically, and dispatching tasks to the Fray engine.

- Tags: internals
- Published: 2026-08-28

### [Prerequisites for Running Marin: From Local CPU to Distributed GPU Clusters](/marin-community/marin/what-are-the-prerequisites-for-running-marin)

Discover the essential prerequisites for running Marin, from local CPU setups to distributed GPU clusters. Learn about Python 3.12+, uv, Git, and hardware requirements.

- Tags: getting-started
- Published: 2026-08-27

### [How to Report a Bug in Marin: A Step-by-Step Guide to Contributing Issues](/marin-community/marin/how-to-report-a-bug-in-marin)

Learn how to report a bug in the marin-community/marin repository effectively. Follow our step-by-step guide to create detailed bug reports and contribute to Marin's development.

- Tags: how-to-guide
- Published: 2026-08-27

### [History of the Marin Project: From Stanford Research to Open-Source LLM Platform](/marin-community/marin/what-is-the-history-of-the-marin-project)

Explore the Marin project history from Stanford research to an open-source LLM platform. Learn how it evolved into a modular system for training large language models.

- Tags: getting-started
- Published: 2026-08-27

### [Where to Find Documentation for Marin: A Complete Guide to Tutorials, API Reference, and Developer Resources](/marin-community/marin/where-can-i-find-documentation-for-marin)

Find comprehensive documentation for Marin, including tutorials, API references, and developer resources, hosted on ReadTheDocs and within the marin-community/marin repository.

- Tags: api-reference
- Published: 2026-08-27

### [Marin Performance Considerations: Scaling Laws, Architecture, and Hardware Tuning](/marin-community/marin/what-are-the-performance-considerations-for-marin)

Explore Marin performance considerations including IsoFLOP scaling, decoupled architecture, efficient storage, tokenization caching, and hardware XLA tuning for optimal results.

- Tags: performance
- Published: 2026-08-27

### [How to Deploy Marin to Production: A Complete Guide to Pulumi-Based Rollouts](/marin-community/marin/how-to-deploy-marin-to-production)

Deploy Marin to production using Pulumi-based rollouts with the marin-deploy CLI or GitHub Actions. Follow this guide for a seamless service deployment.

- Tags: how-to-guide
- Published: 2026-08-27

### [Best Practices for Using the Marin Framework: A Complete Guide](/marin-community/marin/what-are-the-best-practices-for-using-marin)

Master Marin best practices for reproducible pipelines. Leverage lazy artifacts, explicit versioning, and immutable data classes for efficient language model development.

- Tags: best-practices
- Published: 2026-08-27

### [How to Debug Issues in Marin: A Complete Troubleshooting Guide](/marin-community/marin/how-to-debug-issues-in-marin)

Debug Marin issues effectively. Use Iris CLI to inspect scheduler, query the controller database, profile tasks, and roll back checkpoints when Marin becomes unresponsive.

- Tags: how-to-guide
- Published: 2026-08-27

### [Development Roadmap for Marin: Scalable ML Pipeline Framework Goals and Milestones](/marin-community/marin/what-is-the-development-roadmap-for-marin)

Explore the Marin development roadmap focusing on scalable ML pipelines. Discover our goals for multi-node JAX integration, unified observability, and expanded data connectors. Contribute to the future of Marin.

- Tags: getting-started
- Published: 2026-08-27

### [How to Integrate Marin with Other Services: A Step-by-Step Guide](/marin-community/marin/how-to-integrate-marin-with-other-services)

Integrate Marin with other services using adapter classes and SweepConfig API. Follow this step-by-step guide to connect Marin to your existing workflows.

- Tags: how-to-guide
- Published: 2026-08-27

### [Example Applications Using Marin: Tutorials and Sample Scripts for ML Pipelines](/marin-community/marin/are-there-any-example-applications-using-marin)

Explore example applications using Marin to build lazy artifact pipelines, device-agnostic model training, and distributed ML workflows. Get tutorials and sample scripts now.

- Tags: tutorial
- Published: 2026-08-27

### [What Is the License for the Marin Project?](/marin-community/marin/what-is-the-license-for-the-marin-project)

Discover the Marin project license. Learn about Apache License 2.0, its permissive usage rights, and patent protection for your projects.

- Tags: getting-started
- Published: 2026-08-27

### [How to Contribute to the Marin Project: A Complete Developer’s Guide](/marin-community/marin/how-to-contribute-to-the-marin-project)

Learn how to contribute to the Marin project. Follow our developer's guide to set up your environment, understand architecture rules, and submit successful pull requests for marin-community/marin.

- Tags: how-to-guide
- Published: 2026-08-27

### [Configuration Options for Marin: A Complete Guide to the ML Pipeline Framework](/marin-community/marin/what-configuration-options-are-available-for-marin)

Explore Marin's configuration options for ML pipelines. Master inference, training, and data processing with immutable dataclasses for seamless model serving. Get the complete guide.

- Tags: how-to-guide
- Published: 2026-08-27

### [How the Marin API Works: Architecture and Endpoints Explained](/marin-community/marin/how-does-the-marin-api-work)

Explore the Marin API architecture and endpoints. Learn how it converts JSON payloads into dataclasses and orchestrates pipeline execution via a modular step runner. Understand the Marin API's inner workings.

- Tags: architecture
- Published: 2026-08-27

### [What Is the marin-node Module? Fleet Observability in Marin](/marin-community/marin/what-is-the-purpose-of-the-marin-node-module)

Discover the marin-node module for Marin observability. Gain insights into cluster capacity and per-node details with Grafana-powered visualization for your entire fleet.

- Tags: deep-dive
- Published: 2026-08-27

### [How to Run Tests in the Marin Repository: Complete Developer Guide](/marin-community/marin/how-to-run-tests-in-the-marin-repository)

Learn how to run tests in the Marin repository using uv run pytest for quick checks or the comprehensive test suite with safe defaults. Your complete developer guide.

- Tags: how-to-guide
- Published: 2026-08-27

### [What Are the Dependencies for Marin? Complete Guide to the Python Workspace](/marin-community/marin/what-are-the-dependencies-for-marin)

Explore Marin's dependencies in this comprehensive guide. Understand internal packages like marin-iris and external libraries like watchdog, all managed via pyproject.toml.

- Tags: how-to-guide
- Published: 2026-08-27

### [How to Build Marin from Source: Complete Installation Guide](/marin-community/marin/how-to-build-marin-from-source)

Build Marin from source with this complete installation guide. Follow steps to clone, set up your environment, install dependencies, and configure variables for a successful build.

- Tags: getting-started
- Published: 2026-08-27

### [Where to Find the Main Entry Point for the Marin Application](/marin-community/marin/where-can-i-find-the-main-entry-point-for-the-marin-application)

Locate the Marin application's main entry point at marin-serve, which directs to the main() function in iris_cli.py. Find the core logic for Marin here.

- Tags: how-to-guide
- Published: 2026-08-27

### [Marin Project Structure: A Deep Dive into the Monorepo Architecture](/marin-community/marin/how-is-the-marin-project-structured)

Explore the marin project structure, a Python monorepo housing libraries, experiments, infra, and docs. Discover its core data processing and distributed utilities.

- Tags: deep-dive
- Published: 2026-08-27

### [What Programming Languages Are Used in Marin? A Polyglot Codebase Analysis](/marin-community/marin/what-programming-languages-are-used-in-marin)

Explore the Marin repository's polyglot codebase. Discover how Python, Rust, JavaScript, and Shell scripts power its machine learning, services, documentation, and automation.

- Tags: code-analysis
- Published: 2026-08-27

### [Core Components of the Marin Project: A Deep Dive into the LLM Training Stack](/marin-community/marin/what-are-the-core-components-of-the-marin-project)

Explore the core components of the Marin Project's LLM training stack: Levanter, Iris, Zephyr, Fray, Rigging, and Marin Core. Discover a complete JAX-based solution for efficient model training.

- Tags: deep-dive
- Published: 2026-08-27

### [How to Set Up Marin for Local Development: Complete Installation Guide](/marin-community/marin/how-to-set-up-marin-for-local-development)

Easily set up Marin for local development with this complete installation guide. Follow simple steps to clone the repository, install dependencies, and run the tiny-model tutorial.

- Tags: getting-started
- Published: 2026-08-27