How DBX Handles Large Result Sets with Virtual Scrolling: Implementation Guide

DBX handles large result sets by combining Vue's RecycleScroller component for DOM virtualization, page-sized data fetching capped at 1,000 rows, and infinite-scroll detection that automatically loads additional data when users approach the viewport bottom.

The t8y2/dbx repository implements a high-performance data grid capable of displaying massive datasets without browser lag. By leveraging virtual scrolling and intelligent pagination, DBX ensures that only visible rows consume memory and DOM resources, even when querying tables with millions of records.

Virtual Scrolling Architecture

The RecycleScroller Component

In apps/desktop/src/components/grid/DataGrid.vue, DBX integrates the vue-virtual-scroller library's <RecycleScroller> component to maintain constant memory usage regardless of result set size. This component renders only the rows visible in the viewport plus a small buffer, recycling DOM elements as the user scrolls rather than creating thousands of row nodes.

<RecycleScroller
  ref="gridScroller"
  class="data-grid-scroller"
  :items="displayItems"
  :item-size="30"
  :buffer="400"
  key-field="id"
  @scroll="onScroll"
>
  <template #default="{ item }">
    <div class="data-grid-row" :style="{ height: '30px' }">
      {{ item.display }}
    </div>
  </template>
</RecycleScroller>

The configuration uses a fixed item-size of 30 pixels to match the CSS row height, ensuring accurate scroll position calculations. The 400-pixel buffer pre-renders rows just outside the visible area, preventing flickering during rapid scrolling.

Row Rendering Strategy

By binding :items="displayItems" to the current page of data rather than the entire result set, the grid keeps DOM node count constant. As users scroll through large tables, the scroller efficiently updates the content of existing row elements rather than mounting new components, which prevents the memory bloat typically associated with large data grids.

Pagination Controls and Limits

Page Size Configuration

In apps/desktop/src/lib/dataGrid/dataGridPagination.ts, DBX defines strict boundaries to prevent excessive memory consumption and network payload issues:

export const MAX_RESULT_PAGE_SIZE = 1000;
export const MIN_RESULT_PAGE_SIZE = 1;

export function normalizeResultPageSize(pageSize: number): number {
  return Math.max(MIN_RESULT_PAGE_SIZE, Math.min(MAX_RESULT_PAGE_SIZE, Math.round(pageSize)));
}

export const resultPageSizeMenuOptions = [
  { value: 50, label: "50 rows" },
  { value: 100, label: "100 rows (default)" },
  { value: 250, label: "250 rows" },
  { value: 500, label: "500 rows" },
  { value: 1000, label: "1000 rows" },
];

The default page size is 100 rows, but users can select up to 1,000 rows per request. The normalizeResultPageSize function ensures that any user input or stored preference stays within the safe range of 1 to 1,000 rows.

Detecting Available Data

The canGoNextDataGridPage function determines whether additional pages exist by comparing the current row count against the page size:

export function canGoNextDataGridPage({ rowCount, pageSize }: { rowCount: number; pageSize: number }): boolean {
  return rowCount === pageSize;
}

When a query returns exactly the requested page size, DBX assumes more data may exist and enables pagination controls. If the result contains fewer rows than the page size, the grid recognizes it has reached the end of the dataset.

Infinite Scroll Implementation

Scroll Detection Logic

The apps/desktop/src/lib/dataGrid/dataGridInfiniteScroll.ts file provides utility functions that calculate scroll metrics to determine when users approach the bottom of the current data page. The isDataGridNearScrollBottom helper analyzes the scroll position, scroll height, and client height to trigger pagination before the user reaches the absolute bottom of the viewport.

The Pagination Trigger

When the user scrolls, the onScroll handler in DataGrid.vue coordinates between the scroll detection utilities and the pagination state:

function onScroll(e: Event) {
  const el = e.target as HTMLElement;
  const metrics = {
    scrollTop: el.scrollTop,
    scrollHeight: el.scrollHeight,
    clientHeight: el.clientHeight,
  };

  if (isDataGridNearScrollBottom(metrics) && canGoNextDataGridPage({
    rowCount: props.result.rows.length,
    pageSize: settingsStore.editorSettings.pageSize,
  })) {
    const nextOffset = currentOffset + settingsStore.editorSettings.pageSize;
    emit('paginate', nextOffset, settingsStore.editorSettings.pageSize);
  }
}

This handler prevents unnecessary network requests by checking both scroll position and data availability before emitting the paginate event. The component also tracks isScrolling state to avoid triggering pagination during active cell editing or drag operations.

State Management and Persistence

DBX persists the user's preferred page size across sessions using the settingsStore.editorSettings.pageSize property. When the application loads, it retrieves the stored preference and normalizes it through normalizeResultPageSize to ensure compatibility with current limits. The query store then uses this value when building SQL queries with offset and limit parameters:

function loadMore(offset: number, limit: number) {
  const sql = buildDataGridCountSql(...);
  api.runQuery({ sql, offset, limit }).then(result => {
    displayItems.value.push(...result.rows);
  });
}

Summary

• Virtual DOM Recycling: The <RecycleScroller> component in DataGrid.vue maintains constant memory usage by rendering only visible rows with a 400-pixel buffer.
• Configurable Page Limits: The pagination system enforces a maximum of 1,000 rows per request through MAX_RESULT_PAGE_SIZE and normalizeResultPageSize.
• Smart Pagination Triggers: The canGoNextDataGridPage function combined with isDataGridNearScrollBottom ensures data loads only when needed and available.
• Persistent Preferences: User-selected page sizes persist across sessions via settingsStore, with safe normalization to prevent invalid values.

Frequently Asked Questions

What is the maximum number of rows DBX can display at once?

While DBX can handle result sets containing millions of rows, the UI fetches and displays data in pages of up to 1,000 rows at a time. This limit, defined in dataGridPagination.ts, prevents browser memory issues and ensures responsive performance. Users can select smaller page sizes (50, 100, 250, or 500) through the settings menu.

How does DBX prevent memory leaks when displaying large tables?

DBX prevents memory leaks through the vue-virtual-scroller library's <RecycleScroller> component, which reuses a fixed pool of DOM elements regardless of dataset size. Instead of creating thousands of row nodes, the grid updates the content of existing elements as users scroll, keeping memory consumption flat even with multi-million row result sets.

Can users customize how many rows load per page?

Yes, users can configure their preferred page size through the resultPageSizeMenuOptions array, which exposes options for 50, 100, 250, 500, and 1,000 rows. This preference persists in settingsStore.editorSettings.pageSize and applies to all subsequent queries. The normalizeResultPageSize function ensures that any custom value falls within the supported range of 1 to 1,000 rows.

What happens when a query returns exactly the page size limit?

When a query returns exactly the number of rows specified by the current page size, the canGoNextDataGridPage function returns true, indicating that additional data may exist. The grid then enables infinite-scroll detection, and if the user scrolls near the bottom, DBX automatically requests the next page using the current offset plus the page size limit.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}