Troubleshooting File Listing in GeoLibre: A Complete Guide to Remote Data Sources

GeoLibre displays remote file listings through a layered plugin architecture that separates metadata retrieval from actual file-list data, with a hard 1,000-item pagination ceiling enforced at the proxy layer.

GeoLibre connects to cloud data repositories like Source Cooperative and Hugging Face through specialized plugins that handle file enumeration. When file listings fail to appear, load incompletely, or trigger infinite loading states, the root cause typically lies in pagination handling, identifier validation, or proxy error propagation. This guide walks through the source code in opengeos/GeoLibre to diagnose and fix these issues.

How GeoLibre File Listings Work

The file listing flow follows a strict three-stage pipeline designed for performance and reliability.

Core Request Flow

When a user adds a remote dataset, the UI invokes a plugin-specific fetch method. For Source Cooperative sources, this is SourceCoopFetch defined in packages/plugins/src/plugins/source-coop-api.ts. The plugin returns a OnePageOfProductFiles structure containing minimal identifiers—accountId, productId, and fileId—without full metadata to keep payloads lightweight.

The comment at line 165 in source-coop-api.ts explicitly states this design choice: the file listing drives the UI, so the system deliberately avoids pulling complete metadata for each entry. Only upon user selection does GeoLibre fetch full metadata via a separate endpoint.

Proxy Layer Enforcement

All remote calls route through the worker proxy in packages/plugins/src/plugins/maplibre-source-coop.ts. This proxy performs three critical functions:

1. Authentication header injection
2. Page-size ceiling enforcement (1,000 items maximum)
3. Error translation into UI-friendly messages

The pagination limit originates from the S3-compatible API backend. The constant PAGE_SIZE defined at line 102 in source-coop-api.ts carries the comment: /** S3 page size for a file listing. The proxy's own ceiling is 1000. */

Common File Listing Failures and Fixes

Symptom	Likely Cause	Verification Step	Solution
No files appear after adding source	Invalid accountId/productId pair	Inspect DevTools Network for GET to …/files?pageSize=1000; check query string contains correct IDs	Re-enter source URL or select correct product in UI
Exactly 1,000 items shown when more exist	Pagination stopped after first page	Look for subsequent requests with pageToken or continuationToken; absence indicates missing fetch logic	Update MapLibreSourceCoop component to trigger fetchNextPage() on scroll or button click
"Loading…" spinner persists indefinitely	Proxy error (e.g., 429) swallowed by UI	Check console.error in maplibre-source-coop.ts around line 495 for failed fetch details	Implement exponential backoff or increase proxy rate-limit configuration
Files visible but click returns "File not found"	File identifier missing required accountId/productId	Verify request payload near line 1015 comment includes these fields	Amend source-coop-api.ts export or update calling component to pass identifiers

Debugging File Listing Issues Step by Step

Step 1: Verify API Response Payload

Open browser DevTools → Network tab. Locate the request to the Source Cooperative endpoint. Confirm the JSON response contains files: [...] array. Empty or malformed arrays indicate upstream API problems before GeoLibre code executes.

Step 2: Check Pagination Token Handling

If the response includes nextPageToken, trace whether the UI consumes it. The OnePageOfProductFiles interface in source-coop-api.ts defines this field—its presence without subsequent requests signals a pagination logic gap.

Step 3: Review Proxy Error Logs

The proxy's error handling (lines 495-520 in maplibre-source-coop.ts) logs detailed failure messages. Reproduce the issue and examine console output for HTTP status codes, rate-limit headers, or credential problems.

Step 4: Confirm UI State Transitions

In the React component rendering the list (conventionally FileList.tsx or similar within apps/geolibre-desktop/src/components/), verify that "Load more" interactions:

• Increment an internal page counter
• Pass the pageToken to the fetch method
• Merge results correctly with existing state

Step 5: Validate Constants Alignment

The PAGE_SIZE constant must match the server's enforced limit. If the upstream API changes its ceiling, update line 102 in source-coop-api.ts accordingly. Mismatched values cause silent truncation or unnecessary request fragmentation.

Code Examples for File Listing Implementation

Fetching the First Page

import { fetchProductFiles } from "@/packages/plugins/src/plugins/source-coop-api";

const PAGE_SIZE = 1000; // must match proxy ceiling per source-coop-api.ts

async function loadFirstPage(accountId: string, productId: string) {
  const page = await fetchProductFiles({
    accountId,
    productId,
    pageSize: PAGE_SIZE,
  });
  // page.files contains up to 1000 minimal file descriptors
  console.log(`Retrieved ${page.files.length} files`);
  return page;
}

Implementing Pagination in UI Components

import { useState } from 'react';
import { fetchProductFiles, FileInfo } from "@/packages/plugins/src/plugins/source-coop-api";

const PAGE_SIZE = 1000;

function FileListViewer({ accountId, productId }: { accountId: string; productId: string }) {
  const [files, setFiles] = useState<FileInfo[]>([]);
  const [nextToken, setNextToken] = useState<string | null>(null);
  const [isLoading, setIsLoading] = useState(false);

  async function loadMore() {
    if (!nextToken || isLoading) return;
    
    setIsLoading(true);
    try {
      const nextPage = await fetchProductFiles({
        accountId,
        productId,
        pageSize: PAGE_SIZE,
        pageToken: nextToken,
      });
      setFiles(prev => [...prev, ...nextPage.files]);
      setNextToken(nextPage.nextPageToken ?? null);
    } finally {
      setIsLoading(false);
    }
  }

  // Render list and load-more trigger...
}

Key Source Files for Troubleshooting

File Path	Role
packages/plugins/src/plugins/source-coop-api.ts	API contract, PAGE_SIZE constant, OnePageOfProductFiles interface, fetchProductFiles implementation
packages/plugins/src/plugins/maplibre-source-coop.ts	Worker proxy, authentication, error logging (lines 495-520), request forwarding
packages/plugins/src/plugins/huggingface-api.ts	Parallel implementation for Hugging Face sources with different pagination limits
packages/plugins/src/plugins/remote-file-formats.ts	Size limits and format validation rules affecting file browsing panels
apps/geolibre-desktop/src/components/FileList.tsx (typical location)	UI component consuming paginated API and rendering scrollable list

Summary

• GeoLibre file listings rely on a plugin-per-source architecture with strict separation between listing and metadata endpoints
• The 1,000-item page ceiling is hardcoded in source-coop-api.ts and enforced by the proxy in maplibre-source-coop.ts
• Pagination tokens must be explicitly passed and consumed; their absence in network traffic indicates UI logic gaps
• Proxy error messages provide diagnostic detail but require console inspection since they're often not surfaced to the UI
• Minimal identifier payloads optimize listing performance—full metadata fetches occur only on user selection

Frequently Asked Questions

Why does my Source Cooperative listing show exactly 1,000 files when the repository has more?

GeoLibre's PAGE_SIZE constant in source-coop-api.ts aligns with the S3-compatible API's maximum page size. The first page returns 1,000 items with a nextPageToken that your UI component must use to request subsequent pages. If loading stops, verify your component calls fetchNextPage() or equivalent and passes the token correctly.

Where do I find detailed error messages when file listings fail silently?

Check the browser console for errors logged by maplibre-source-coop.ts between lines 495-520. The proxy catches upstream failures, logs them via console.error, and returns generic messages to the UI. Rate limits (HTTP 429), authentication failures (401), and malformed responses all appear here with full stack traces.

Can I increase the 1,000-item limit for faster bulk loading?

No. This limit originates from the Source Cooperative API infrastructure, not GeoLibre's choice. The comment in source-coop-api.ts explicitly describes it as "the proxy's own ceiling." Attempting larger pageSize values results in rejected requests or truncated responses. Implement proper pagination with pageToken handling instead.

Why do some files appear in the list but fail when clicked?

The listing endpoint returns minimal identifiers, but the detail/metadata endpoint requires complete accountId/productId/fileId tuples. If your UI component passes partial identifiers—as noted near line 1015 in source-coop-api.ts—the subsequent fetch fails. Verify your state management preserves all three fields from the listing response through to the selection handler.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}