# How to Integrate GeoLibre File Listing into Custom Scripts: A Developer's Guide

> Integrate GeoLibre file listing into your custom scripts using listProductFiles or listObjects. Access Source Cooperative datasets programmatically with this developer guide.

- Repository: [Open Geospatial Solutions/GeoLibre](https://github.com/opengeos/GeoLibre)
- Tags: how-to-guide
- Published: 2026-08-18

---

**Import the DOM-free [`source-coop-api.ts`](https://github.com/opengeos/GeoLibre/blob/main/source-coop-api.ts) module, inject any `fetch`-compatible function, and use `listProductFiles()` or `listObjects()` to paginate through Source Cooperative datasets.**

GeoLibre treats every external data source as a **plugin** conforming to a lightweight, framework-free contract. The Source Cooperative plugin at [`packages/plugins/src/plugins/source-coop-api.ts`](https://github.com/opengeos/GeoLibre/blob/main/packages/plugins/src/plugins/source-coop-api.ts) provides a portable client for the S3-style file-listing API used by <https://source.coop>. This article explains how to integrate GeoLibre file listing into custom scripts running in Node.js, browsers, or hybrid environments like Tauri.

## Core Architecture of the Source Cooperative Plugin

The plugin separates **metadata** operations from **data** retrieval through two distinct base URLs:

```ts
// From packages/plugins/src/plugins/source-coop-api.ts (lines 93-101)
export const SOURCE_COOP_API_BASE = "https://source.coop/api/v1";
export const SOURCE_COOP_DATA_BASE = "https://data.source.coop";
export const SOURCE_COOP_PROXY_ENDPOINT = "https://tiles.geolibre.app/source-coop";

```

The **metadata endpoint** lacks CORS headers and serves JSON API responses. The **data endpoint** is CORS-enabled for direct browser access. GeoLibre's worker proxy at `tiles.geolibre.app/source-coop` bridges these environments when needed.

### Pagination Safety Limits

GeoLibre caps each request well below the Worker's 1,000-key hard limit:

```ts
// From packages/plugins/src/plugins/source-coop-api.ts (lines 102-104)
export const SOURCE_COOP_LIST_MAX_KEYS = 200; // page size used for each request

```

This balance minimizes memory pressure while reducing total API calls for large datasets.

## Low-Level File Listing with listObjects()

The `listObjects()` function builds S3 `ListObjectsV2` requests with proper query parameter encoding:

```ts
// From packages/plugins/src/plugins/source-coop-api.ts (lines 150-180)
export async function listObjects(
  fetch: SourceCoopFetch,
  accountId: string,
  productId: string,
  token?: string,
): Promise<SourceCoopListing> {
  const url = new URL(`${SOURCE_COOP_DATA_BASE}/${accountId}/${productId}`);
  url.searchParams.set('list-type', '2');
  url.searchParams.set('max-keys', String(SOURCE_COOP_LIST_MAX_KEYS));
  if (token) url.searchParams.set('continuation-token', token);

  const response = await fetch(url.toString());
  const body = await response.text();
  const json = JSON.parse(body);

  return {
    objects: json.Contents?.map((c: any) => ({
      key: c.Key,
      name: c.Key.split('/').pop() ?? c.Key,
      size: Number(c.Size),
      lastModified: c.LastModified ?? null,
      format: classifyKey(c.Key),
      url: `${SOURCE_COOP_DATA_BASE}/${c.Key}`,
    })) ?? [],
    folders: json.CommonPrefixes?.map((p: any) => p.Prefix) ?? [],
    nextToken: json.NextContinuationToken ?? null,
  };
}

```

**Key design decisions** in this implementation:
- **URL construction** uses `URL.searchParams` to safely encode `?prefix=` filters for subfolder traversal
- **Defensive parsing** handles the Worker's JSON guarantee while remaining resilient to edge cases
- **Format classification** delegates to `classifyKey()` from [`remote-file-formats.ts`](https://github.com/opengeos/GeoLibre/blob/main/remote-file-formats.ts) for consistent type detection

## Convenient Async Iteration with listProductFiles()

For most use cases, `listProductFiles()` wraps pagination logic in a clean async generator:

```ts
// From packages/plugins/src/plugins/source-coop-api.ts (lines 190-210)
export async function* listProductFiles(
  fetch: SourceCoopFetch,
  accountId: string,
  productId: string,
): AsyncGenerator<SourceCoopObject> {
  let token: string | undefined;
  do {
    const page = await listObjects(fetch, accountId, productId, token);
    for (const obj of page.objects) yield obj;
    token = page.nextToken ?? undefined;
  } while (token);
}

```

This abstraction follows continuation tokens automatically, yielding individual file objects until the listing exhausts.

## Practical Integration Examples

### Example 1: Simple One-Off Listing

Use `defaultFetch` for immediate results in Node.js or browsers:

```ts
import { listProductFiles, defaultFetch } from
  "geo-libre/packages/plugins/src/plugins/source-coop-api";

const ACCOUNT = "protomaps";
const PRODUCT = "openstreetmap";

(async () => {
  for await (const file of listProductFiles(defaultFetch, ACCOUNT, PRODUCT)) {
    console.log(`${file.name}\t${file.size} bytes\t${file.format}`);
  }
})();

```

### Example 2: Manual Pagination Control

Access raw pages when you need folder boundaries or progress tracking:

```ts
import { listObjects, defaultFetch } from "geo-libre/.../source-coop-api";

(async () => {
  let token: string | undefined;
  do {
    const page = await listObjects(defaultFetch, "protomaps", "openstreetmap", token);
    console.log(`🗂️ ${page.objects.length} files, ${page.folders.length} folders`);
    page.objects.forEach(o => console.log(`- ${o.name} (${o.size} B)`));
    token = page.nextToken ?? undefined;
  } while (token);
})();

```

### Example 3: Streaming Pipeline with Format Filtering

Combine listing with GeoLibre's format utilities for selective processing:

```ts
import {
  listProductFiles,
  defaultFetch,
  isTooLargeToOpen,
  canStream,
} from "geo-libre/.../source-coop-api";

(async () => {
  for await (const file of listProductFiles(defaultFetch, "protomaps", "openstreetmap")) {
    if (canStream(file.format) && !isTooLargeToOpen(file.size)) {
      // Stream to DuckDB-WASM, MapLibre, or custom consumers
      console.log(`→ streaming ${file.name}`);
    } else {
      console.log(`⚠️ skip ${file.name} (unsupported or too large)`);
    }
  }
})();

```

The `classifyKey` function (re-exported from [`remote-file-formats.ts`](https://github.com/opengeos/GeoLibre/blob/main/remote-file-formats.ts)) enables this logic without duplicating format detection code.

## File Type Classification and Utilities

The Source Cooperative plugin re-exports format classification tools from [`packages/plugins/src/plugins/remote-file-formats.ts`](https://github.com/opengeos/GeoLibre/blob/main/packages/plugins/src/plugins/remote-file-formats.ts):

```ts
// From packages/plugins/src/plugins/source-coop-api.ts (lines 60-71)
export { classifyPath as classifyKey, ... } from "./remote-file-formats";

```

These utilities categorize keys by extension and content signatures, exposing properties like:
- **Streaming compatibility** — whether the format supports byte-range requests
- **Size thresholds** — heuristics for "too large to open" warnings
- **Driver mapping** — which GeoLibre renderer handles each format

## Environment Flexibility and Custom Fetch Injection

The DOM-free design accepts any **`fetch`-compatible function**:

| Environment | Typical injection | Notes |
|-------------|-------------------|-------|
| Node.js 18+ | Native `fetch` or `undici` | `defaultFetch` auto-detects |
| Browser | Native `fetch` | Routes through `SOURCE_COOP_PROXY_ENDPOINT` when CORS requires |
| Tauri desktop | `tauri/http` client | Bypasses CORS entirely via native stack |
| Testing | Mock `fetch` | Inject for deterministic fixtures |

This portability explains how the same [`source-coop-api.ts`](https://github.com/opengeos/GeoLibre/blob/main/source-coop-api.ts) module serves GeoLibre's web build, desktop application, and server-side tooling without modification.

## Key Files for Reference

| File | Purpose | Direct Link |
|------|---------|-------------|
| [`packages/plugins/src/plugins/source-coop-api.ts`](https://github.com/opengeos/GeoLibre/blob/main/packages/plugins/src/plugins/source-coop-api.ts) | Core client — URL building, pagination, typed responses | [source-coop-api.ts](https://github.com/opengeos/GeoLibre/blob/main/packages/plugins/src/plugins/source-coop-api.ts) |
| [`packages/plugins/src/plugins/remote-file-formats.ts`](https://github.com/opengeos/GeoLibre/blob/main/packages/plugins/src/plugins/remote-file-formats.ts) | Format catalogue and size limits | [remote-file-formats.ts](https://github.com/opengeos/GeoLibre/blob/main/packages/plugins/src/plugins/remote-file-formats.ts) |
| [`packages/plugins/src/plugins/maplibre-source-coop.ts`](https://github.com/opengeos/GeoLibre/blob/main/packages/plugins/src/plugins/maplibre-source-coop.ts) | UI integration layer for MapLibre rendering | [maplibre-source-coop.ts](https://github.com/opengeos/GeoLibre/blob/main/packages/plugins/src/plugins/maplibre-source-coop.ts) |

## Summary

- **GeoLibre file listing integration** requires only a `fetch` function and the [`source-coop-api.ts`](https://github.com/opengeos/GeoLibre/blob/main/source-coop-api.ts) module — no DOM or UI dependencies
- **`listProductFiles()`** provides the easiest API via async iteration, while **`listObjects()`** exposes raw pagination control
- **Dual-base URL architecture** separates metadata and data access patterns, with automatic proxy fallback for CORS-restricted environments
- **Format classification utilities** from [`remote-file-formats.ts`](https://github.com/opengeos/GeoLibre/blob/main/remote-file-formats.ts) enable intelligent file filtering without external dependencies
- The **200-key page size** balances API efficiency against memory pressure for large datasets

## Frequently Asked Questions

### How do I handle authentication for private Source Cooperative repositories?

The current `listObjects()` signature accepts an optional token parameter, but as implemented in [`source-coop-api.ts`](https://github.com/opengeos/GeoLibre/blob/main/source-coop-api.ts) at lines 150-180, this refers to S3 **continuation tokens** for pagination rather than authentication credentials. Source Cooperative's public datasets require no authentication; for private access, you would need to extend the `fetch` injection to include `Authorization` headers with your Source Cooperative API key, then pass this custom fetch to `listProductFiles()` or `listObjects()`.

### Can I use GeoLibre file listing in a server-side Node.js script without the full GeoLibre package?

Yes — the [`source-coop-api.ts`](https://github.com/opengeos/GeoLibre/blob/main/source-coop-api.ts) module is deliberately **DOM-free** and framework-agnostic. Import only the plugin package or vendor the single file; the only runtime dependency is a `fetch`-compatible function. Node.js 18+ provides native `fetch`, or you can inject `undici`, `node-fetch`, or any other implementation matching the `SourceCoopFetch` type signature.

### What happens when a dataset contains more than 1,000 files?

GeoLibre's **automatic pagination** in `listProductFiles()` follows `NextContinuationToken` values until exhaustion. The 200-key page size (defined at line 102 of [`source-coop-api.ts`](https://github.com/opengeos/GeoLibre/blob/main/source-coop-api.ts)) ensures each request stays well under Source Cooperative's Worker limit of 1,000 keys. For datasets with tens of thousands of objects, consider adding your own throttling or parallelization around the generator interface.

### How do I filter listings to specific subdirectories?

Pass a `prefix` query parameter by modifying the URL construction. The current implementation at lines 157-163 in [`source-coop-api.ts`](https://github.com/opengeos/GeoLibre/blob/main/source-coop-api.ts) uses `url.searchParams.set` for standard parameters; you would add `url.searchParams.set('prefix', 'your/subfolder/')` before the fetch call. For a reusable solution, wrap `listObjects()` with a custom function that accepts and injects the prefix parameter.