How to Integrate GeoLibre File Listing into Custom Scripts: A Developer's Guide

Import the DOM-free source-coop-api.ts module, inject any fetch-compatible function, and use listProductFiles() or listObjects() to paginate through Source Cooperative datasets.

GeoLibre treats every external data source as a plugin conforming to a lightweight, framework-free contract. The Source Cooperative plugin at packages/plugins/src/plugins/source-coop-api.ts provides a portable client for the S3-style file-listing API used by https://source.coop. This article explains how to integrate GeoLibre file listing into custom scripts running in Node.js, browsers, or hybrid environments like Tauri.

Core Architecture of the Source Cooperative Plugin

The plugin separates metadata operations from data retrieval through two distinct base URLs:

// From packages/plugins/src/plugins/source-coop-api.ts (lines 93-101)
export const SOURCE_COOP_API_BASE = "https://source.coop/api/v1";
export const SOURCE_COOP_DATA_BASE = "https://data.source.coop";
export const SOURCE_COOP_PROXY_ENDPOINT = "https://tiles.geolibre.app/source-coop";

The metadata endpoint lacks CORS headers and serves JSON API responses. The data endpoint is CORS-enabled for direct browser access. GeoLibre's worker proxy at tiles.geolibre.app/source-coop bridges these environments when needed.

Pagination Safety Limits

GeoLibre caps each request well below the Worker's 1,000-key hard limit:

// From packages/plugins/src/plugins/source-coop-api.ts (lines 102-104)
export const SOURCE_COOP_LIST_MAX_KEYS = 200; // page size used for each request

This balance minimizes memory pressure while reducing total API calls for large datasets.

Low-Level File Listing with listObjects()

The listObjects() function builds S3 ListObjectsV2 requests with proper query parameter encoding:

// From packages/plugins/src/plugins/source-coop-api.ts (lines 150-180)
export async function listObjects(
  fetch: SourceCoopFetch,
  accountId: string,
  productId: string,
  token?: string,
): Promise<SourceCoopListing> {
  const url = new URL(`${SOURCE_COOP_DATA_BASE}/${accountId}/${productId}`);
  url.searchParams.set('list-type', '2');
  url.searchParams.set('max-keys', String(SOURCE_COOP_LIST_MAX_KEYS));
  if (token) url.searchParams.set('continuation-token', token);

  const response = await fetch(url.toString());
  const body = await response.text();
  const json = JSON.parse(body);

  return {
    objects: json.Contents?.map((c: any) => ({
      key: c.Key,
      name: c.Key.split('/').pop() ?? c.Key,
      size: Number(c.Size),
      lastModified: c.LastModified ?? null,
      format: classifyKey(c.Key),
      url: `${SOURCE_COOP_DATA_BASE}/${c.Key}`,
    })) ?? [],
    folders: json.CommonPrefixes?.map((p: any) => p.Prefix) ?? [],
    nextToken: json.NextContinuationToken ?? null,
  };
}

Key design decisions in this implementation:

• URL construction uses URL.searchParams to safely encode ?prefix= filters for subfolder traversal
• Defensive parsing handles the Worker's JSON guarantee while remaining resilient to edge cases
• Format classification delegates to classifyKey() from remote-file-formats.ts for consistent type detection

Convenient Async Iteration with listProductFiles()

For most use cases, listProductFiles() wraps pagination logic in a clean async generator:

// From packages/plugins/src/plugins/source-coop-api.ts (lines 190-210)
export async function* listProductFiles(
  fetch: SourceCoopFetch,
  accountId: string,
  productId: string,
): AsyncGenerator<SourceCoopObject> {
  let token: string | undefined;
  do {
    const page = await listObjects(fetch, accountId, productId, token);
    for (const obj of page.objects) yield obj;
    token = page.nextToken ?? undefined;
  } while (token);
}

This abstraction follows continuation tokens automatically, yielding individual file objects until the listing exhausts.

Practical Integration Examples

Example 1: Simple One-Off Listing

Use defaultFetch for immediate results in Node.js or browsers:

import { listProductFiles, defaultFetch } from
  "geo-libre/packages/plugins/src/plugins/source-coop-api";

const ACCOUNT = "protomaps";
const PRODUCT = "openstreetmap";

(async () => {
  for await (const file of listProductFiles(defaultFetch, ACCOUNT, PRODUCT)) {
    console.log(`${file.name}\t${file.size} bytes\t${file.format}`);
  }
})();

Example 2: Manual Pagination Control

Access raw pages when you need folder boundaries or progress tracking:

import { listObjects, defaultFetch } from "geo-libre/.../source-coop-api";

(async () => {
  let token: string | undefined;
  do {
    const page = await listObjects(defaultFetch, "protomaps", "openstreetmap", token);
    console.log(`🗂️ ${page.objects.length} files, ${page.folders.length} folders`);
    page.objects.forEach(o => console.log(`- ${o.name} (${o.size} B)`));
    token = page.nextToken ?? undefined;
  } while (token);
})();

Example 3: Streaming Pipeline with Format Filtering

Combine listing with GeoLibre's format utilities for selective processing:

import {
  listProductFiles,
  defaultFetch,
  isTooLargeToOpen,
  canStream,
} from "geo-libre/.../source-coop-api";

(async () => {
  for await (const file of listProductFiles(defaultFetch, "protomaps", "openstreetmap")) {
    if (canStream(file.format) && !isTooLargeToOpen(file.size)) {
      // Stream to DuckDB-WASM, MapLibre, or custom consumers
      console.log(`→ streaming ${file.name}`);
    } else {
      console.log(`⚠️ skip ${file.name} (unsupported or too large)`);
    }
  }
})();

The classifyKey function (re-exported from remote-file-formats.ts) enables this logic without duplicating format detection code.

File Type Classification and Utilities

The Source Cooperative plugin re-exports format classification tools from packages/plugins/src/plugins/remote-file-formats.ts:

// From packages/plugins/src/plugins/source-coop-api.ts (lines 60-71)
export { classifyPath as classifyKey, ... } from "./remote-file-formats";

These utilities categorize keys by extension and content signatures, exposing properties like:

• Streaming compatibility — whether the format supports byte-range requests
• Size thresholds — heuristics for "too large to open" warnings
• Driver mapping — which GeoLibre renderer handles each format

Environment Flexibility and Custom Fetch Injection

The DOM-free design accepts any fetch-compatible function:

Environment	Typical injection	Notes
Node.js 18+	Native fetch or undici	defaultFetch auto-detects
Browser	Native fetch	Routes through SOURCE_COOP_PROXY_ENDPOINT when CORS requires
Tauri desktop	tauri/http client	Bypasses CORS entirely via native stack
Testing	Mock fetch	Inject for deterministic fixtures

This portability explains how the same source-coop-api.ts module serves GeoLibre's web build, desktop application, and server-side tooling without modification.

Key Files for Reference

File	Purpose	Direct Link
packages/plugins/src/plugins/source-coop-api.ts	Core client — URL building, pagination, typed responses	source-coop-api.ts
packages/plugins/src/plugins/remote-file-formats.ts	Format catalogue and size limits	remote-file-formats.ts
packages/plugins/src/plugins/maplibre-source-coop.ts	UI integration layer for MapLibre rendering	maplibre-source-coop.ts

Summary

• GeoLibre file listing integration requires only a fetch function and the source-coop-api.ts module — no DOM or UI dependencies
• listProductFiles() provides the easiest API via async iteration, while listObjects() exposes raw pagination control
• Dual-base URL architecture separates metadata and data access patterns, with automatic proxy fallback for CORS-restricted environments
• Format classification utilities from remote-file-formats.ts enable intelligent file filtering without external dependencies
• The 200-key page size balances API efficiency against memory pressure for large datasets

Frequently Asked Questions

How do I handle authentication for private Source Cooperative repositories?

The current listObjects() signature accepts an optional token parameter, but as implemented in source-coop-api.ts at lines 150-180, this refers to S3 continuation tokens for pagination rather than authentication credentials. Source Cooperative's public datasets require no authentication; for private access, you would need to extend the fetch injection to include Authorization headers with your Source Cooperative API key, then pass this custom fetch to listProductFiles() or listObjects().

Can I use GeoLibre file listing in a server-side Node.js script without the full GeoLibre package?

Yes — the source-coop-api.ts module is deliberately DOM-free and framework-agnostic. Import only the plugin package or vendor the single file; the only runtime dependency is a fetch-compatible function. Node.js 18+ provides native fetch, or you can inject undici, node-fetch, or any other implementation matching the SourceCoopFetch type signature.

What happens when a dataset contains more than 1,000 files?

GeoLibre's automatic pagination in listProductFiles() follows NextContinuationToken values until exhaustion. The 200-key page size (defined at line 102 of source-coop-api.ts) ensures each request stays well under Source Cooperative's Worker limit of 1,000 keys. For datasets with tens of thousands of objects, consider adding your own throttling or parallelization around the generator interface.

How do I filter listings to specific subdirectories?

Pass a prefix query parameter by modifying the URL construction. The current implementation at lines 157-163 in source-coop-api.ts uses url.searchParams.set for standard parameters; you would add url.searchParams.set('prefix', 'your/subfolder/') before the fetch call. For a reusable solution, wrap listObjects() with a custom function that accepts and injects the prefix parameter.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}