How Marin EndpointsService Enables Service Discovery and Lazy Resolution

The Marin EndpointsService enables service discovery and lazy resolution by maintaining a pluggable registry of scheme resolvers that convert abstract URIs (such as gcp:// or k8s://) into concrete HTTP URLs only when resolve_endpoint_uri is called, eliminating network overhead during configuration parsing.

The Marin distributed system framework decouples service configuration from physical infrastructure using a flexible endpoints system. Unlike traditional service discovery that resolves addresses during startup, the Marin EndpointsService (iris.cluster.endpoints) implements lazy resolution to keep cluster initialization fast and avoid unnecessary external network calls.

Core Architecture of the Marin EndpointsService

The endpoints system centers on a lightweight resolution layer that abstracts multiple backend providers behind a uniform API. This design allows cluster configurations to declare logical service locations using scheme-tagged URIs while deferring the actual lookup until runtime.

The Scheme Resolver Registry

At the heart of the service lies a private registry mapping scheme names to resolver callables. In lib/iris/src/iris/cluster/endpoints.py, the _REGISTRY dictionary stores these mappings, populated via the public register_scheme function:

def register_scheme(scheme: str, handler: SchemeResolver) -> None:
    """Register ``handler`` as the resolver for ``scheme://...`` URIs."""
    _REGISTRY[scheme] = handler          # → https://github.com/marin-community/marin/blob/main/lib/iris/src/iris/cluster/endpoints.py#L57-L60

Built-in resolvers for Google Cloud Platform (gcp://) and Kubernetes (k8s://) are registered automatically at module import time on lines 94‑95. This Registry pattern makes the system scheme-agnostic and allows new discovery mechanisms to be added without modifying core controller code.

URI Resolution API

The primary entry point for all resolution requests is resolve_endpoint_uri (line 62), which accepts a scheme-tagged URI and metadata dictionary, then returns a concrete http(s):// URL:

from iris.cluster.endpoints import resolve_endpoint_uri

uri = "gcp://my-controller-vm"
metadata = {"zone": "us-central1-a", "port": "10001"}
concrete_url = resolve_endpoint_uri(uri, metadata)

# → "http://10.128.0.5:10001"

How Lazy Resolution Works in Marin

Lazy resolution means that resolver functions are invoked only when a component explicitly requests the concrete address. During configuration parsing, no network calls occur—not even DNS lookups. This defers expensive operations like gcloud CLI executions or Kubernetes API queries until the actual moment the URL is needed.

The controller triggers this resolution via _resolve_cluster_endpoints in lib/iris/src/iris/cluster/controller/main.py (lines 46‑61). This function iterates over cluster_config.endpoints and calls resolve_endpoint_uri for each entry, ensuring that startup remains fast while guaranteeing that components receive resolvable addresses before they attempt connections.

Built-in Scheme Resolvers

Marin ships with three primary resolvers that handle the most common deployment scenarios.

GCP Cloud VM Discovery

The _resolve_gcp function (lines 27‑55) handles gcp:// URIs by constructing and executing a gcloud command to retrieve the instance's internal IP address. It parses the CLI output and constructs the final URL:

# Example resolution flow for gcp://my-vm

# 1. Executes: gcloud compute instances describe my-vm --zone=us-central1-a

# 2. Extracts internal IP: 10.128.0.5

# 3. Returns: http://10.128.0.5:10001

Kubernetes In-Cluster Resolution

For k8s:// URIs, the _resolve_k8s resolver (lines 67‑91) performs local DNS construction without external API calls. It transforms the URI into the standard Kubernetes cluster DNS format:

# k8s://service-name/namespace resolves to:

http://service-name.namespace.svc.cluster.local:port

This approach avoids requiring kubeconfig or API access from the controller, relying instead on the cluster's internal DNS infrastructure.

Direct HTTP/HTTPS Passthrough

URIs using http:// or https:// schemes bypass resolution entirely. The resolve_endpoint_uri function returns these unchanged, allowing configurations to mix static URLs with dynamic discovery endpoints seamlessly.

Extending Service Discovery with Custom Schemes

The registration API enables developers to inject custom discovery logic without forking the core framework. Any callable matching the SchemeResolver signature can handle new protocols:

from iris.cluster.endpoints import register_scheme, SchemeResolver

def myproto_resolver(uri: str, meta: dict[str, str]) -> str:
    # Custom logic to resolve the endpoint

    host = uri.split("://", 1)[1]
    return f"https://myproxy.example.com/{host}"

register_scheme("myproto", myproto_resolver)

# Usage

print(resolve_endpoint_uri("myproto://serviceA", {}))

# → "https://myproxy.example.com/serviceA"

Once registered, the custom resolver participates fully in lazy resolution alongside built-in providers.

Integration with the Cluster Controller

The controller coordinates endpoint discovery during cluster initialization. In lib/iris/src/iris/cluster/controller/main.py, the _resolve_cluster_endpoints function (lines 46‑61) consumes the configuration and populates a dictionary of resolved addresses:

# Inside iris.cluster.controller.main

endpoints = _resolve_cluster_endpoints(cluster_config)
log_server = endpoints.get("/system/log-server", "")

# `log_server` now holds the concrete URL or an empty string if not configured.

This integration point demonstrates how Marin separates configuration schema from runtime resolution, allowing the same configuration to resolve differently across development, staging, and production environments without code changes.

Summary

• Marin EndpointsService provides scheme-agnostic service discovery through a pluggable resolver registry stored in lib/iris/src/iris/cluster/endpoints.py.
• Lazy resolution defers all network operations until resolve_endpoint_uri is explicitly called, keeping startup performance consistent regardless of backend latency.
• Built-in resolvers handle Google Cloud VMs (gcp://), Kubernetes services (k8s://), and static HTTP/HTTPS URLs without requiring external configuration.
• Extensible registration via register_scheme allows custom protocols to integrate seamlessly with the existing resolution pipeline.
• Controller integration in lib/iris/src/iris/cluster/controller/main.py orchestrates bulk resolution while maintaining the lazy evaluation pattern.

Frequently Asked Questions

What is lazy resolution in Marin?

Lazy resolution is a design pattern in the Marin EndpointsService where network-intensive operations—such as querying Google Cloud APIs or performing DNS lookups—are deferred until a component actually requests the concrete service URL. This ensures that configuration parsing and system startup remain fast and do not block on external dependencies.

How do I add a custom scheme resolver?

Import register_scheme from iris.cluster.endpoints and pass your resolver function along with the scheme name. Your function must accept a URI string and metadata dictionary and return a concrete HTTP or HTTPS URL. Once registered, any configuration using your scheme (e.g., custom://service) will automatically route to your resolver when resolve_endpoint_uri is called.

When does Marin actually resolve gcp:// and k8s:// endpoints?

Resolution occurs at runtime when the controller or a specific component invokes resolve_endpoint_uri. For the cluster controller, this happens inside _resolve_cluster_endpoints during initialization, but after the configuration has been loaded. No resolution occurs during the initial parsing of the YAML or JSON configuration files.

What happens if an endpoint URI scheme is not registered?

If resolve_endpoint_uri encounters a scheme that does not exist in the _REGISTRY, the function will raise a KeyError or similar exception indicating that no resolver is available for that scheme. Valid configurations should only use schemes that are either built-in (http, https, gcp, k8s) or explicitly registered via register_scheme before resolution is attempted.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}