DBX Virtual-Scrolled Data Grid Architecture for Large Result Sets: Technical Implementation Guide

DBX implements a virtual-scroll architecture that renders only visible rows while maintaining a full-height invisible gutter, enabling millions of rows to be browsed with constant memory usage and seamless infinite-scroll pagination.

The t8y2/dbx open-source database client solves the browser memory bottleneck when displaying massive SQL result sets through a sophisticated virtual-scrolled data grid architecture. By decoupling the scrollable area from actual DOM nodes and calculating viewport windows on the fly, the grid maintains native 60fps performance regardless of whether the underlying dataset contains hundreds or millions of rows.

Core Architectural Components

Scroll Position and Metrics Tracking

The foundation of the virtual-scroll system lives in apps/desktop/src/lib/dataGrid/dataGridInfiniteScroll.ts. This module exports DataGridScrollPosition and DataGridScrollMetrics, which capture the current vertical and horizontal scroll offsets and provide helper utilities to determine when additional data is required. These metrics calculate the exact pixel position within the theoretical full dataset, allowing the grid to map scroll coordinates to row indices without loading all rows into memory.

Infinite-Scroll Trigger Detection

The same file contains the critical trigger functions isDataGridNearScrollBottom and shouldCheckInfiniteScrollAfterScroll. These utilities compare the current scroll metrics against configurable thresholds (defaulting to 100px) to determine when the user has scrolled close enough to the bottom of the currently rendered content to warrant fetching the next page. This detection runs efficiently on scroll events without causing layout thrashing.

Scroll Gutter and Viewport Simulation

apps/desktop/src/lib/dataGrid/dataGridScrollGutter.ts implements the invisible "gutter" element—a dummy <div> sized to rowCount * rowHeight. This placeholder occupies the full theoretical height of the result set, ensuring the browser's native scrollbar reflects the correct proportional position and total size. The actual data rows are rendered as a floating layer on top of this gutter, translated vertically to match the scroll position.

Data Loading and Pagination Logic

When the infinite-scroll trigger fires, apps/desktop/src/lib/dataGrid/dataGridPagination.ts orchestrates the asynchronous page request. It merges newly fetched rows into the existing in-memory array and updates the gutter height to reflect the growing dataset. This module ensures that the rows[] array grows only as needed, while the scroll bar automatically adjusts to represent the updated total row count.

Selection and Clipboard Utilities

Despite only rendering a subset of rows, selection operations work against the full logical dataset. apps/desktop/src/lib/dataGrid/gridSelection.ts exports extractSelection, summarizeSelection, and formatSelectionAsCsv (along with TSV, JSON, and SQL variants). These functions operate directly on the complete in-memory row array, ensuring that copy-and-paste, CSV export, and aggregate calculations remain accurate even when most rows are not currently mounted in the DOM.

Decoupled Column State Management

To prevent unnecessary re-renders of the massive data payload, column configuration is isolated in separate modules. apps/desktop/src/lib/dataGrid/dataGridColumnOrder.ts, apps/desktop/src/lib/dataGrid/dataGridColumnVisibility.ts, and dataGridColumnWidth.ts persist column order, visibility flags, and width values independently from row data. This separation allows the grid to recompute visual layouts instantly when users resize or reorder columns without touching the underlying result set.

How the Virtual-Scroll Pipeline Works

The architecture follows a precise seven-step lifecycle:

1. Initial Load — The grid requests the first page of rows (typically 100–200 records) and stores them in an internal rows[] array.

2. Gutter Height Calculation — Using the total row count from query metadata, the system calculates the full dataset height and sets the scroll gutter element's height to rowCount * rowHeight, creating a scrollable area matching the theoretical full result set.

3. Viewport Determination — On each scroll event, the grid computes firstVisibleRow = Math.floor(scrollTop / rowHeight) and lastVisibleRow = firstVisibleRow + visibleRowCount. Only rows within this window are rendered.

4. Infinite-Scroll Check — The system invokes isDataGridNearScrollBottom(metrics, threshold) after each scroll. When the user scrolls within the threshold distance of the bottom, it triggers a pagination request.

5. Data Append — New rows are appended to the existing rows[] array. Because the gutter already reflects the full row count, the scroll bar automatically adjusts to the correct proportional size without jumping.

6. Selection and Export — Functions like extractSelection and formatSelectionAsCsv operate on the complete rows[] array, ensuring that copying or exporting data captures the full logical selection regardless of current viewport visibility.

7. Column Layout — Column-related changes apply instantly via separate state objects, allowing the grid to repaint headers and cell widths without re-processing the massive data payload.

Performance Characteristics and Scalability Benefits

Memory Efficiency — Only the rows currently visible in the viewport (typically 20–50 DOM nodes) exist in the document at any moment, keeping memory usage constant regardless of total result set size.

CPU Efficiency — React reconciliation runs exclusively on the small visible subset. The bulk of the data resides in plain JavaScript arrays, minimizing component lifecycle overhead and garbage collection pressure.

Smooth User Experience — The decoupled scroll gutter provides a native scrollbar that behaves intuitively, scrolling fluidly even while new data is being fetched in the background. Users perceive the full dataset as immediately available.

Modular Design — Each concern (scroll metrics, infinite-scroll triggers, pagination, column handling, selection logic) resides in isolated modules. This separation simplifies unit testing and enables future extensions such as server-side sorting or virtual column resizing.

Practical Implementation Examples

The following snippets demonstrate how the virtual-scroll logic integrates into the DBX grid component:

// Scroll handler with infinite-scroll detection
import { 
  dataGridScrollPosition, 
  shouldCheckInfiniteScrollAfterScroll, 
  isDataGridNearScrollBottom 
} from './dataGridInfiniteScroll';
import { requestNextPage } from '../services/api';

function onScroll(e: UIEvent) {
  const target = e.target as HTMLElement;
  const current = dataGridScrollPosition(target.scrollTop, target.scrollLeft);

  if (shouldCheckInfiniteScrollAfterScroll(prevScrollRef.current, current)) {
    const metrics = {
      scrollTop: target.scrollTop,
      scrollHeight: target.scrollHeight,
      clientHeight: target.clientHeight,
    };
    
    if (isDataGridNearScrollBottom(metrics)) {
      requestNextPage(); // async append to rows[]
    }
  }

  prevScrollRef.current = current;
}

// Rendering only the visible viewport slice
function renderRows(
  rows: GridCellValue[][], 
  scrollTop: number, 
  rowHeight: number, 
  visibleCount: number
) {
  const first = Math.floor(scrollTop / rowHeight);
  const last = Math.min(rows.length, first + visibleCount);
  const slice = rows.slice(first, last);

  return slice.map((row, i) => (
    <tr key={first + i}>
      {row.map((cell, j) => (
        <td key={j}>{cell ?? ''}</td>
      ))}
    </tr>
  ));
}

// Exporting selection from the full dataset
import { extractSelection, formatSelectionAsCsv } from './gridSelection';

function exportSelectionAsCsv(
  columns: string[], 
  rows: GridCellValue[][], 
  range: CellSelectionRange
) {
  const selection = extractSelection(columns, rows, range);
  const csv = formatSelectionAsCsv(selection);
  download(csv, 'selection.csv', 'text/csv');
}

Summary

• Virtual-scroll architecture renders only visible rows while maintaining a full-height scroll gutter in apps/desktop/src/lib/dataGrid/dataGridScrollGutter.ts.
• Infinite-scroll detection uses isDataGridNearScrollBottom and shouldCheckInfiniteScrollAfterScroll from dataGridInfiniteScroll.ts to trigger pagination.
• Memory efficiency is achieved by keeping only viewport rows in the DOM, with the full dataset stored in plain JavaScript arrays.
• Selection integrity is preserved via extractSelection and formatSelectionAsCsv in gridSelection.ts, which operate on the complete logical dataset.
• Modular column state in dataGridColumnOrder.ts and dataGridColumnVisibility.ts allows layout changes without data re-renders.

Frequently Asked Questions

How does DBX handle selection and export for rows that are not currently rendered?

Selection logic operates on the full in-memory rows[] array rather than the DOM. Functions like extractSelection and formatSelectionAsCsv in apps/desktop/src/lib/dataGrid/gridSelection.ts accept the complete dataset and selection range, returning the correct data regardless of whether the rows are currently mounted in the viewport. This ensures that copying to clipboard or exporting to CSV captures the entire selection even when most rows remain virtualized.

What triggers the next page load in the infinite-scroll system?

The isDataGridNearScrollBottom function in apps/desktop/src/lib/dataGrid/dataGridInfiniteScroll.ts evaluates scroll metrics after each scroll event. When the user's scroll position falls within a configurable threshold (typically 100 pixels) of the bottom of the scroll gutter, the function returns true, triggering the pagination logic in dataGridPagination.ts to fetch the next chunk of rows.

How does the scroll gutter maintain correct scrollbar behavior without loading all rows?

apps/desktop/src/lib/dataGrid/dataGridScrollGutter.ts creates an invisible placeholder div sized to the total height of the result set (rowCount * rowHeight). This element sits behind the actual grid content and provides the browser's native scrollbar with the correct track size and thumb position. As the user scrolls, the visible rows are rendered as an overlay positioned absolutely to match the scroll offset, creating the illusion of a fully loaded list while only a handful of DOM nodes exist.

Can the virtual grid handle horizontal scrolling for result sets with many columns?

Yes. The architecture treats column management separately from row virtualization through modules like apps/desktop/src/lib/dataGrid/dataGridColumnOrder.ts and dataGridColumnVisibility.ts. Column widths, order, and visibility are stored in independent state objects, allowing the grid to compute horizontal scroll offsets and render only visible cells without affecting the vertical virtual-scroll logic or requiring re-fetching of row data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}