GeoLibre Repository File Listing API: A Complete Developer's Guide

The GeoLibre repository file listing API is a DOM-free, S3-compatible module in packages/plugins/src/plugins/source-coop-api.ts that fetches paginated geospatial datasets from cloud providers like Source Coop, Hugging Face, and OpenAerialMap.

This article explains how the GeoLibre repository file listing API works under the hood. You'll learn the exact data structures, pagination strategy, and code patterns used to browse remote geospatial data—knowledge drawn directly from the opengeos/GeoLibre source code.

Architecture of the File Listing API

The API follows a layered design that keeps fetching, parsing, and UI concerns separate.

Layer	Responsibility	Key Source File
Remote fetch abstraction	Supplies a simple fetch(url, signal?) wrapper for testability	source-coop-api.ts – SourceCoopFetch & defaultFetch
Data model	TypeScript types for products, files, and paginated listings	SourceCoopProduct, SourceCoopObject, SourceCoopListing
Constants	Page limits, base URLs, and CORS-enabled proxy endpoints	SOURCE_COOP_API_BASE, SOURCE_COOP_PROXY_ENDPOINT, SOURCE_COOP_LIST_MAX_KEYS
Parsing helpers	JSON normalization, safe extraction, "account/product" parsing	asString, asRecord, parseProductRef
Listing logic	Build S3-style ListObjectsV2 URLs, request pages, convert responses	listObjects function
UI glue	Component that calls listing functions and routes to format handlers	maplibre-source-coop.ts

This same structure is reused across huggingface-api.ts and openaerialmap-api.ts, making the GeoLibre repository file listing API consistent across providers.

Core Constants and Configuration

The module hardcodes environment-specific URLs and limits. These constants control where requests go and how large each page can be.

export const SOURCE_COOP_API_BASE        = "https://source.coop/api/v1";      // No CORS → desktop only
export const SOURCE_COOP_DATA_BASE       = "https://data.source.coop";        // Direct data, CORS-enabled
export const SOURCE_COOP_PROXY_ENDPOINT  = "https://tiles.geolibre.app/source-coop";
export const SOURCE_COOP_LIST_MAX_KEYS   = 200;  // Proxy ceiling (worker caps at 1000)

Key insight: The proxy endpoint (tiles.geolibre.app/source-coop) exists because api/v1 lacks CORS headers. Browser-based requests must route through this allowlisted worker.

Data Types: What the API Returns

The GeoLibre repository file listing API uses strictly typed interfaces to represent files and listings.

export interface SourceCoopObject {
  key: string;               // S3 key includes "{productId}/" prefix
  name: string;              // Final path segment for UI display
  size: number;              // Bytes
  lastModified: string | null;
  format: SourceCoopFormat;  // Re-exported from remote-file-formats.ts
  url: string;               // Browser-fetchable URL on data.source.coop
}

export interface SourceCoopListing {
  objects: SourceCoopObject[];
  folders: string[];         // Sub-folder prefixes (S3 CommonPrefixes)
  nextToken: string | null;  // Pagination cursor
}

The nextToken field enables lazy-loaded, scroll-aware browsing—the UI never loads the full directory tree.

Parsing Product References

Users enter account/product strings in the search box. The parseProductRef function validates and splits these with a strict RegExp:

const PRODUCT_REF_RE = /^([a-z0-9][a-z0-9-_.]*)\/([a-z0-9][a-z0-9-_.]*)\/?$/i;

export function parseProductRef(query: string) {
  const match = PRODUCT_REF_RE.exec(query.trim());
  return match 
    ? { accountId: match[1], productId: match[2] } 
    : null;
}

Invalid patterns return null, letting the UI show immediate feedback without network requests.

Building S3-Compatible Listing URLs

The GeoLibre repository file listing API hides S3's query-parameter quirks. Two URL patterns are explicitly invalid:

• /{account}/ → returns 404 (NoSuchBucket)
• /{account}/{product}/{sub}/?list-type=2 → returns 400 (InvalidRequest)

Sub-folders must use ?prefix= instead. The URL builder handles this:

function buildListingUrl(
  accountId: string,
  productId: string,
  token: string | null,
): string {
  const base = `${SOURCE_COOP_PROXY_ENDPOINT}/list/${accountId}/${productId}`;
  const params = new URLSearchParams({ 
    "list-type": "2", 
    "max-keys": String(SOURCE_COOP_LIST_MAX_KEYS) 
  });
  if (token) params.set("continuation-token", token);
  return `${base}?${params}`;
}

The listObjects Function: Core Implementation

The heart of the GeoLibre repository file listing API is the listObjects async function. It orchestrates fetching, parsing, and normalization.

export async function listObjects(
  accountId: string,
  productId: string,
  fetcher: SourceCoopFetch = defaultFetch,
  token: string | null = null,
): Promise<SourceCoopListing> {
  const url = buildListingUrl(accountId, productId, token);
  const resp = await fetcher(url);
  
  if (!resp.ok) {
    throw new Error(`Failed to list ${accountId}/${productId}: ${resp.status}`);
  }
  
  const text = await resp.text();
  const data = JSON.parse(text);  // S3 XML pre-converted to JSON by worker
  
  const objects = (data.Contents ?? []).map((c: any) => ({
    key: c.Key,
    name: c.Key.split("/").pop() ?? "",
    size: Number(c.Size),
    lastModified: c["LastModified"] ?? null,
    format: classifyPath(c.Key),  // from remote-file-formats.ts
    url: `${SOURCE_COOP_DATA_BASE}/${accountId}/${c.Key}`,
  }));
  
  const folders = (data.CommonPrefixes ?? []).map((p: any) => p.Prefix);
  const nextToken = data.NextContinuationToken ?? null;
  
  return { objects, folders, nextToken };
}

Critical details:

• The fetcher parameter defaults to defaultFetch but can be stubbed in tests
• S3 XML is pre-converted to JSON by the Cloudflare worker at tiles.geolibre.app
• classifyPath() (from remote-file-formats.ts) determines if a file is raster, vector, or streamable

Pagination Strategy for Infinite Scrolling

The UI component in maplibre-source-coop.ts implements cursor-based pagination:

1. Call listObjects without a token for the first page
2. Store nextToken in component state
3. When user scrolls, call listObjects again with the stored token
4. Append new objects to the existing list
5. Repeat until nextToken is null

This pattern keeps memory usage bounded regardless of dataset size.

Code Examples

Basic Usage in Node.js or Test Environment

import { listObjects, parseProductRef } from "./source-coop-api.ts";

// Parse user-entered reference
const ref = "protomaps/openstreetmap";
const ids = parseProductRef(ref);
if (!ids) throw new Error("Invalid reference");

// Fetch first page
const page1 = await listObjects(ids.accountId, ids.productId);
console.log("Files:", page1.objects.map(o => o.name));

// Fetch subsequent pages
if (page1.nextToken) {
  const page2 = await listObjects(
    ids.accountId, 
    ids.productId, 
    undefined, 
    page1.nextToken
  );
  console.log("More files:", page2.objects.map(o => o.name));
}

React Component Integration

import { useEffect, useState } from "react";
import { listObjects, parseProductRef, SourceCoopObject } from "@geolibre/plugins";

export function SourceCoopBrowser({ query }: { query: string }) {
  const [files, setFiles] = useState<SourceCoopObject[]>([]);
  const [nextToken, setNextToken] = useState<string | null>(null);
  const [loading, setLoading] = useState(false);

  useEffect(() => {
    const ids = parseProductRef(query);
    if (!ids) return;
    
    setLoading(true);
    listObjects(ids.accountId, ids.productId)
      .then(page => {
        setFiles(page.objects);
        setNextToken(page.nextToken);
        setLoading(false);
      });
  }, [query]);

  const loadMore = async () => {
    if (!nextToken) return;
    setLoading(true);
    
    const ids = parseProductRef(query)!;
    const page = await listObjects(
      ids.accountId, 
      ids.productId, 
      undefined, 
      nextToken
    );
    
    setFiles(prev => [...prev, ...page.objects]);
    setNextToken(page.nextToken);
    setLoading(false);
  };

  return (
    <div>
      <ul>
        {files.map(f => (
          <li key={f.key}>{f.name} ({f.size.toLocaleString()} bytes)</li>
        ))}
      </ul>
      {nextToken && (
        <button onClick={loadMore} disabled={loading}>
          {loading ? "Loading..." : "Load more"}
        </button>
      )}
    </div>
  );
}

Unit Test with Stubbed Fetch

import { listObjects, SourceCoopFetch } from "./source-coop-api.ts";
import { assertEquals } from "@std/assert";

const fakeFetch: SourceCoopFetch = async (url) => ({
  ok: true,
  status: 200,
  text: async () => JSON.stringify({
    Contents: [{
      Key: "openstreetmap/roads.geojson",
      Size: 12345,
      LastModified: "2024-01-01T00:00:00Z"
    }],
    CommonPrefixes: [],
    NextContinuationToken: null,
  }),
});

Deno.test("listObjects returns parsed object", async () => {
  const listing = await listObjects("protomaps", "openstreetmap", fakeFetch);
  
  assertEquals(listing.objects.length, 1);
  assertEquals(listing.objects[0].name, "roads.geojson");
  assertEquals(listing.objects[0].size, 12345);
  assertEquals(listing.nextToken, null);
});

Reuse Across Cloud Providers

The GeoLibre repository file listing API pattern extends to other plugins:

Provider	File	Key Constant	Endpoint Pattern
Source Coop	source-coop-api.ts	SOURCE_COOP_LIST_MAX_KEYS = 200	/list/{account}/{product}
Hugging Face	huggingface-api.ts	HF_PAGE_SIZE = 1000	/tree/{rev}/{path}
OpenAerialMap	openaerialmap-api.ts	(varies)	/meta service

All three re-export types from remote-file-formats.ts for consistent format classification.

Key Source Files

File	Purpose
packages/plugins/src/plugins/source-coop-api.ts	Core implementation: types, listObjects, parseProductRef
packages/plugins/src/plugins/remote-file-formats.ts	Shared classifyPath() and format detection
packages/plugins/src/plugins/maplibre-source-coop.ts	UI component that consumes the listing API
packages/plugins/src/plugins/huggingface-api.ts	Parallel implementation for Hugging Face Hub
workers/tiles/src/allowlisted-fetch.ts	CORS proxy enforcing allow-lists for all remote fetches

Summary

• The GeoLibre repository file listing API lives in packages/plugins/src/plugins/source-coop-api.ts and provides paginated, S3-compatible directory listings
• Cursor-based pagination via nextToken enables infinite scrolling without loading entire datasets
• Injectable fetcher (SourceCoopFetch) makes the API testable without network dependencies
• Strict product reference parsing with parseProductRef validates account/product strings before any network call
• CORS proxy at tiles.geolibre.app bridges browser restrictions when direct API access lacks headers
• Identical API shape across Source Coop, Hugging Face, and OpenAerialMap plugins simplifies multi-provider UIs

Frequently Asked Questions

How does the GeoLibre repository file listing API handle pagination?

The API returns a nextToken string when more results exist. Pass this token as the fourth argument to listObjects() to fetch the next page. When nextToken is null, you've reached the end of the listing. The UI component maplibre-source-coop.ts uses this pattern to implement lazy-loaded infinite scrolling.

Can I use the file listing API outside of a browser?

Yes. The fetcher parameter in listObjects() accepts any function matching the SourceCoopFetch signature. Pass node-fetch or Deno's native fetch in server or test environments. The default defaultFetch binds to the global fetch, which works in modern Node.js versions.

Why does the API use a proxy endpoint instead of calling Source Coop directly?

The Source Coop API base (https://source.coop/api/v1) does not send CORS headers, which blocks browser requests. The proxy at https://tiles.geolibre.app/source-coop adds the necessary headers and enforces an allow-list. This is implemented in workers/tiles/src/allowlisted-fetch.ts as a Cloudflare Worker.

How are file formats detected in the listing response?

The classifyPath() function from remote-file-formats.ts examines each file's key (path) and returns a SourceCoopFormat enum value. This determines whether GeoLibre can stream, render as raster, or open the file as a vector layer. The same classification logic is shared across all remote-browse plugins.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}