# What Is the Marin‑Ducky Service? A DuckDB‑Powered SQL Dashboard for Cloud Storage

> Explore the Marin-Ducky service a DuckDB-powered SQL dashboard for querying data in Google Cloud Storage and other object stores. Run ad-hoc SQL queries easily.

- Repository: [The Marin Project/marin](https://github.com/marin-community/marin)
- Tags: getting-started
- Published: 2026-09-10

---

**Marin‑Ducky is an ad‑hoc DuckDB‑backed SQL service that provides a lightweight, web‑based dashboard for running SQL queries against data stored in Google Cloud Storage (GCS) and other object stores.**

The **Marin‑ducky service** is part of the marin‑community/marin repository and offers data engineers a fast, temporary way to explore datasets without deploying a full‑blown analytics platform. Built on the **Starlette** web framework and integrated with the Marin → Iris job orchestration layer, it launches as an ephemeral Iris job that inherits port naming and service registration from the orchestration layer.

## Core Architecture and Components

The service is modular by design, separating concerns between the web interface, query execution, configuration, and auditing.

### Dashboard UI and Starlette Integration

The user interface is defined in [`lib/ducky/src/ducky/server.py`](https://github.com/marin-community/marin/blob/main/lib/ducky/src/ducky/server.py) and serves a simple HTML page where users paste SQL and execute it. The UI is served under a proxy path such as `/proxy/ducky/` to coexist with other services behind a reverse proxy. The `create_app()` function initializes the Starlette application, while `serve()` wires the app to an Iris‑provided port and registers the service name (`ducky`) with the Iris supervisor.

### Query Execution with QueryRunner

Queries are executed by the **QueryRunner** class located in [`lib/ducky/src/ducky/runner.py`](https://github.com/marin-community/marin/blob/main/lib/ducky/src/ducky/runner.py). This component invokes DuckDB, writes results to a temporary "scratch" bucket (for example, `gs://marin-ducky-us-east5/ducky/...`), and returns a `QueryResult` object containing the schema, rows, and metadata. Results are persisted as Parquet files in GCS, allowing downstream tools to consume them directly.

### Configuration via DuckyConfig

Runtime settings are managed by the `DuckyConfig` class in [`lib/ducky/src/ducky/config.py`](https://github.com/marin-community/marin/blob/main/lib/ducky/src/ducky/config.py). Key parameters include:

- **scratch_bucket** – The GCS path where temporary results are stored.
- **allowed_buckets** – A tuple of permitted read prefixes that the service validates before accessing data.

This configuration ensures the service only touches authorized storage locations.

### Query Logging and Auditing

Every query is recorded in a **QueryLog** ([`lib/ducky/src/ducky/query_log.py`](https://github.com/marin-community/marin/blob/main/lib/ducky/src/ducky/query_log.py)). The log captures timestamps, result paths, status information, and metadata, facilitating later analysis, cost attribution, or debugging of failed queries.

## Deployment and Iris Integration

Marin‑Ducky is packaged as the `marin-ducky` workspace in [`pyproject.toml`](https://github.com/marin-community/marin/blob/main/pyproject.toml), making it installable as a standalone component. The deployment is defined in [`infra/ducky/Pulumi.ducky-marin.yaml`](https://github.com/marin-community/marin/blob/main/infra/ducky/Pulumi.ducky-marin.yaml) and triggered by the *ducky* roll‑out workflow in [`scripts/ci/pulumi_rollouts.py`](https://github.com/marin-community/marin/blob/main/scripts/ci/pulumi_rollouts.py).

When deployed as an Iris job, the service becomes discoverable within the Marin ecosystem. The Iris supervisor handles port allocation and service naming, allowing other Marin components to reach the Ducky dashboard via predictable internal DNS.

## Stateless Design and Fault Tolerance

The **Marin‑ducky service** is intentionally stateless. If the process restarts or is preempted, any running query is lost, but already‑produced result files remain in the scratch bucket and can be reused. This design enables cheap, short‑lived deployments that survive preemptions without data loss, making it ideal for ad‑hoc analytics on spot instances.

## Practical Code Examples

### Running a Query via the Client API

Use the `DuckyClient` to execute SQL programmatically and retrieve result metadata:

```python
from ducky.client import DuckyClient

client = DuckyClient(base_url="http://ducky.test/proxy/ducky")
result = client.query("SELECT COUNT(*) FROM my_table")

print(result.stdout)        # → number of rows

print(result.result_path)   # GCS path to the full result Parquet file

```

### Launching the Server Inside an Iris Job

To start the service within the Marin job orchestration framework:

```python
from ducky.server import create_app, serve

# create_app() returns the Starlette instance

# serve() binds to the Iris-provided port and registers the service

serve()

```

### Configuring a Custom Scratch Bucket

Restrict the service to specific GCS paths for security and cost control:

```python
from ducky.config import DuckyConfig

config = DuckyConfig(
    scratch_bucket="gs://my-custom-bucket/ducky",
    allowed_buckets=("gs://my-custom-bucket",),
)

```

## Summary

- **Marin‑Ducky** is a lightweight, DuckDB‑powered SQL service for exploring data in GCS through a web dashboard.
- It consists of a **Starlette** web app ([`server.py`](https://github.com/marin-community/marin/blob/main/server.py)), a **QueryRunner** for execution ([`runner.py`](https://github.com/marin-community/marin/blob/main/runner.py)), and a **DuckyConfig** for security policies ([`config.py`](https://github.com/marin-community/marin/blob/main/config.py)).
- The service is **stateless** and restart‑safe, storing results in a configurable scratch bucket for fault tolerance.
- It deploys as an **Iris job** via Pulumi, inheriting service discovery and port management from the Marin orchestration layer.
- All queries are audited through the **QueryLog** ([`query_log.py`](https://github.com/marin-community/marin/blob/main/query_log.py)) for debugging and compliance.

## Frequently Asked Questions

### How does Marin‑Ducky handle query results after a process restart?

Because the service is stateless, an in‑flight query is lost if the process restarts. However, any result files already written to the scratch bucket (for example, `gs://marin-ducky-us-east5/ducky/...`) persist and remain accessible. Users can re‑run the query or retrieve the existing Parquet output directly from GCS.

### What web framework powers the Marin‑Ducky dashboard?

The dashboard is built on **Starlette**, an async Python web framework. The entry point in [`lib/ducky/src/ducky/server.py`](https://github.com/marin-community/marin/blob/main/lib/ducky/src/ducky/server.py) defines the routes and the `serve()` function handles integration with the Iris job orchestration layer for dynamic port binding.

### Can I restrict which GCS buckets Marin‑Ducky is allowed to read from?

Yes. The `DuckyConfig` class in [`lib/ducky/src/ducky/config.py`](https://github.com/marin-community/marin/blob/main/lib/ducky/src/ducky/config.py) accepts an `allowed_buckets` tuple. The service validates that all query‑time paths start with one of these prefixes before accessing data, preventing unauthorized cross‑project reads.

### How is the Marin‑ducky service deployed in production?

The service is defined as a Pulumi stack in [`infra/ducky/Pulumi.ducky-marin.yaml`](https://github.com/marin-community/marin/blob/main/infra/ducky/Pulumi.ducky-marin.yaml) and rolled out via [`scripts/ci/pulumi_rollouts.py`](https://github.com/marin-community/marin/blob/main/scripts/ci/pulumi_rollouts.py). It runs as an **Iris job**, which means the Marin orchestrator manages its lifecycle, port allocation, and service registration automatically.