What Is the Marin‑Ducky Service? A DuckDB‑Powered SQL Dashboard for Cloud Storage

Marin‑Ducky is an ad‑hoc DuckDB‑backed SQL service that provides a lightweight, web‑based dashboard for running SQL queries against data stored in Google Cloud Storage (GCS) and other object stores.

The Marin‑ducky service is part of the marin‑community/marin repository and offers data engineers a fast, temporary way to explore datasets without deploying a full‑blown analytics platform. Built on the Starlette web framework and integrated with the Marin → Iris job orchestration layer, it launches as an ephemeral Iris job that inherits port naming and service registration from the orchestration layer.

Core Architecture and Components

The service is modular by design, separating concerns between the web interface, query execution, configuration, and auditing.

Dashboard UI and Starlette Integration

The user interface is defined in lib/ducky/src/ducky/server.py and serves a simple HTML page where users paste SQL and execute it. The UI is served under a proxy path such as /proxy/ducky/ to coexist with other services behind a reverse proxy. The create_app() function initializes the Starlette application, while serve() wires the app to an Iris‑provided port and registers the service name (ducky) with the Iris supervisor.

Query Execution with QueryRunner

Queries are executed by the QueryRunner class located in lib/ducky/src/ducky/runner.py. This component invokes DuckDB, writes results to a temporary "scratch" bucket (for example, gs://marin-ducky-us-east5/ducky/...), and returns a QueryResult object containing the schema, rows, and metadata. Results are persisted as Parquet files in GCS, allowing downstream tools to consume them directly.

Configuration via DuckyConfig

Runtime settings are managed by the DuckyConfig class in lib/ducky/src/ducky/config.py. Key parameters include:

• scratch_bucket – The GCS path where temporary results are stored.
• allowed_buckets – A tuple of permitted read prefixes that the service validates before accessing data.

This configuration ensures the service only touches authorized storage locations.

Query Logging and Auditing

Every query is recorded in a QueryLog (lib/ducky/src/ducky/query_log.py). The log captures timestamps, result paths, status information, and metadata, facilitating later analysis, cost attribution, or debugging of failed queries.

Deployment and Iris Integration

Marin‑Ducky is packaged as the marin-ducky workspace in pyproject.toml, making it installable as a standalone component. The deployment is defined in infra/ducky/Pulumi.ducky-marin.yaml and triggered by the ducky roll‑out workflow in scripts/ci/pulumi_rollouts.py.

When deployed as an Iris job, the service becomes discoverable within the Marin ecosystem. The Iris supervisor handles port allocation and service naming, allowing other Marin components to reach the Ducky dashboard via predictable internal DNS.

Stateless Design and Fault Tolerance

The Marin‑ducky service is intentionally stateless. If the process restarts or is preempted, any running query is lost, but already‑produced result files remain in the scratch bucket and can be reused. This design enables cheap, short‑lived deployments that survive preemptions without data loss, making it ideal for ad‑hoc analytics on spot instances.

Practical Code Examples

Running a Query via the Client API

Use the DuckyClient to execute SQL programmatically and retrieve result metadata:

from ducky.client import DuckyClient

client = DuckyClient(base_url="http://ducky.test/proxy/ducky")
result = client.query("SELECT COUNT(*) FROM my_table")

print(result.stdout)        # → number of rows

print(result.result_path)   # GCS path to the full result Parquet file

Launching the Server Inside an Iris Job

To start the service within the Marin job orchestration framework:

from ducky.server import create_app, serve

# create_app() returns the Starlette instance

# serve() binds to the Iris-provided port and registers the service

serve()

Configuring a Custom Scratch Bucket

Restrict the service to specific GCS paths for security and cost control:

from ducky.config import DuckyConfig

config = DuckyConfig(
    scratch_bucket="gs://my-custom-bucket/ducky",
    allowed_buckets=("gs://my-custom-bucket",),
)

Summary

• Marin‑Ducky is a lightweight, DuckDB‑powered SQL service for exploring data in GCS through a web dashboard.
• It consists of a Starlette web app (server.py), a QueryRunner for execution (runner.py), and a DuckyConfig for security policies (config.py).
• The service is stateless and restart‑safe, storing results in a configurable scratch bucket for fault tolerance.
• It deploys as an Iris job via Pulumi, inheriting service discovery and port management from the Marin orchestration layer.
• All queries are audited through the QueryLog (query_log.py) for debugging and compliance.

Frequently Asked Questions

How does Marin‑Ducky handle query results after a process restart?

Because the service is stateless, an in‑flight query is lost if the process restarts. However, any result files already written to the scratch bucket (for example, gs://marin-ducky-us-east5/ducky/...) persist and remain accessible. Users can re‑run the query or retrieve the existing Parquet output directly from GCS.

What web framework powers the Marin‑Ducky dashboard?

The dashboard is built on Starlette, an async Python web framework. The entry point in lib/ducky/src/ducky/server.py defines the routes and the serve() function handles integration with the Iris job orchestration layer for dynamic port binding.

Can I restrict which GCS buckets Marin‑Ducky is allowed to read from?

Yes. The DuckyConfig class in lib/ducky/src/ducky/config.py accepts an allowed_buckets tuple. The service validates that all query‑time paths start with one of these prefixes before accessing data, preventing unauthorized cross‑project reads.

How is the Marin‑ducky service deployed in production?

The service is defined as a Pulumi stack in infra/ducky/Pulumi.ducky-marin.yaml and rolled out via scripts/ci/pulumi_rollouts.py. It runs as an Iris job, which means the Marin orchestrator manages its lifecycle, port allocation, and service registration automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}