How DuckDB-WASM Handles GeoParquet, Shapefiles, KML, and KMZ Files in GeoLibre

DuckDB-WASM handles GeoParquet, Shapefiles, KML, and KMZ files through a unified SQL pipeline using GDAL's ST_Read function, automatically selecting the correct driver for each format and converting geometries to GeoJSON entirely within the browser.

GeoLibre, an open-source geospatial visualization project maintained by opengeos, leverages DuckDB-WASM to bring server-grade vector data processing to the client side. Unlike traditional web mapping workflows that require backend services or format-specific JavaScript libraries, GeoLibre uses a single abstraction—GDAL's spatial SQL function ST_Read—to ingest diverse file formats. This article explains exactly how the loader works, with direct references to the source code implementation in apps/geolibre-desktop/src/lib/duckdb-vector-loader.ts.

The Unified DuckDB-WASM Loading Pipeline

All four formats follow an identical seven-step pipeline. The consistency of this approach eliminates the need for format-specific parsing code in your application.

Step 1: File Ingestion into the Virtual File System

The DuckDBVectorFile interface represents any vector file to be loaded. Raw bytes are written to DuckDB-WASM's in-memory virtual file system before SQL execution.

For multi-file formats like Shapefile, the siblingFiles array carries required companion files (.dbf, .prj, .shx).

const shpFiles: DuckDbVectorFile = {
  name: "buildings.shp",
  extension: "shp",
  data: await fetch("/data/buildings.shp").then(r => r.arrayBuffer()),
  siblingFiles: [
    { name: "buildings.dbf", extension: "dbf", data: await fetch("/data/buildings.dbf").then(r => r.arrayBuffer()) },
    { name: "buildings.prj", extension: "prj", data: await fetch("/data/buildings.prj").then(r => r.arrayBuffer()) },
  ],
};

Step 2: SQLite Connection Initialization

The loader obtains a temporary database connection via getDatabase()/getSqlDatabase() from duckdb-vector-loader.ts. This connection provides the execution context for GDAL's spatial extension.

Step 3: Automatic GDAL Driver Selection

The ST_Read SQL function automatically selects the appropriate GDAL driver based on file extension and content. No manual driver configuration is required.

• GeoParquet — DuckDB's native Parquet engine handles most processing, with ST_Read providing the spatial metadata layer
• Shapefile — GDAL's ESRI Shapefile driver reads the .shp and companion files from the virtual file system
• KML — GDAL's KML driver parses the XML-based format directly
• KMZ — The same KML driver transparently decompresses the ZIP archive; no manual unzip step needed

Step 4: SQL Query Construction with ST_Read

The loader builds a parameterized query in duckdb-vector-loader.ts. The layerArgSql helper conditionally adds a layer= argument for multi-layer sources.

SELECT * FROM ST_Read('filename', keep_wkb=true [, layer='layer_name'])

The keep_wkb=true argument preserves original Well-Known Binary (WKB) geometries for potential reprojection operations later in the pipeline.

Step 5: Geometry Conversion to GeoJSON

After row retrieval, duckdb-geometry.ts processes the results. The wkbRowsToFeatureCollection function:

1. Identifies the geometry column (aliased to __geolibre_geometry_geojson via the GEOMETRY_JSON_COLUMN constant)
2. Converts WKB bytes to GeoJSON geometry objects
3. Strips GDAL's automatically added OGC_FID column when present

// Result: standard GeoJSON FeatureCollection
{
  "type": "FeatureCollection",
  "features": [
    {"type": "Feature", "geometry": {...}, "properties": {...}},
    ...
  ]
}

Step 6: Error Handling for Unsupported Geometries

The loader implements specific error patterns in duckdb-vector-loader.ts:

• isUnsupportedSurfaceWkbError — Catches Polyhedral Surface and TIN geometry types that DuckDB-WASM cannot yet process
• isGenericUnsupportedWkbError — Handles other WKB parsing failures

These functions enable graceful degradation with user-facing warnings rather than hard failures.

Step 7: Large Dataset Guard

duckdb-vector-guard.ts evaluates estimated feature counts before full loading. For GeoParquet files, it uses the fast read_parquet metadata path to obtain row counts without reading the entire file. Users receive a confirmation prompt when datasets exceed configurable thresholds.

Complete Loading Examples for Each Format

GeoParquet

import { loadVectorFile } from "./duckdb-vector-loader";

const parquetFile: DuckDbVectorFile = {
  name: "roads.parquet",
  extension: "parquet",
  data: await fetch("/data/roads.parquet").then(r => r.arrayBuffer()),
};

const featureCollection = await loadVectorFile(parquetFile);

Shapefile with Sibling Files

const shpFiles: DuckDbVectorFile = {
  name: "buildings.shp",
  extension: "shp",
  data: await fetch("/data/buildings.shp").then(r => r.arrayBuffer()),
  siblingFiles: [
    { name: "buildings.dbf", extension: "dbf", data: await fetch("/data/buildings.dbf").then(r => r.arrayBuffer()) },
    { name: "buildings.prj", extension: "prj", data: await fetch("/data/buildings.prj").then(r => r.arrayBuffer()) },
  ],
};

const featureCollection = await loadVectorFile(shpFiles);

KML

const kmlFile: DuckDbVectorFile = {
  name: "tour.kml",
  extension: "kml",
  data: await fetch("/data/tour.kml").then(r => r.arrayBuffer()),
};

const featureCollection = await loadVectorFile(kmlFile);

KMZ (Compressed KML)

const kmzFile: DuckDbVectorFile = {
  name: "tour.kmz",
  extension: "kmz",
  data: await fetch("/data/tour.kmz").then(r => r.arrayBuffer()),
};

const featureCollection = await loadVectorFile(kmzFile);
// Note: No manual decompression required

All four examples execute internal SQL equivalent to:

SELECT * FROM ST_Read('filename', keep_wkb=true);

Key Source Files and Their Roles

File	Purpose
duckdb-vector-loader.ts	Core orchestration: file system writes, ST_Read query construction, error handling, GeoJSON assembly
duckdb-geometry.ts	Geometry column detection, WKB-to-GeoJSON conversion, OGC_FID cleanup
duckdb-vector-guard.ts	Pre-load size estimation and user confirmation for large datasets
maplibre-vector.ts (plugin)	Extension registration (.geojson, .parquet, .shp, .kml, .kmz) triggering the loader pipeline

Full source: https://github.com/opengeos/GeoLibre

Performance Characteristics

• GeoParquet — Fastest path due to DuckDB's native Parquet support; column pruning and predicate pushdown reduce memory pressure
• Shapefile — Moderate speed; multiple file I/O operations in WASM file system add overhead
• KML/KMZ — Slowest due to XML parsing; KMZ adds minor decompression cost (handled internally by GDAL)

Summary

• DuckDB-WASM provides one SQL interface (ST_Read) for four vector formats through automatic GDAL driver selection
• The DuckDBVectorFile abstraction handles both single files and multi-file archives via the siblingFiles array
• KMZ requires no manual extraction—GDAL's KML driver decompresses transparently
• Geometry conversion preserves original WKB for reprojection, with error handling for unsupported surface types
• Large dataset guards prevent browser memory exhaustion, using fast Parquet metadata queries when available

Frequently Asked Questions

Does DuckDB-WASM require separate code paths for each file format?

No. According to the GeoLibre source code, all formats use the same loadVectorFile function in duckdb-vector-loader.ts. The ST_Read SQL function automatically detects the format and selects the appropriate GDAL driver based on file extension and content signatures.

Can DuckDB-WASM read KMZ files without manual decompression?

Yes. The GDAL KML driver used by ST_Read transparently handles ZIP-compressed KMZ archives. You pass the .kmz file directly to the loader with no unzip step required, as shown in the implementation examples from duckdb-vector-loader.ts.

What happens when a Shapefile is missing its .prj projection file?

The loader will still execute, but geometries may lack proper coordinate reference system information. The ST_Read function proceeds with the data available; downstream code in duckdb-geometry.ts handles the conversion to GeoJSON, though reprojection accuracy depends on having the complete sibling file set including .prj.

How does GeoLibre prevent browser crashes from massive vector files?

duckdb-vector-guard.ts implements a size-checking layer that estimates feature counts before full loading. For GeoParquet, it uses the efficient read_parquet metadata path to obtain row counts instantly. When thresholds are exceeded, the user receives a confirmation dialog rather than an automatic load that could exhaust memory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}