architecture

How Chat2DB Handles Large File Data Import and Export: Streaming Architecture Explained

July 26, 2026 OtterMind/Chat2DB ↗

Chat2DB processes massive datasets by streaming rows in configurable batches of 100,000 records, using asynchronous contexts and pluggable import strategies to maintain constant memory usage regardless of file size.

Chat2DB, an open-source database management tool maintained by OtterMind, implements a sophisticated streaming architecture to handle multi-gigabyte data transfers without exhausting system memory. Unlike conventional tools that load entire files into RAM, Chat2DB processes data incrementally through batch handlers and result-set consumers. This article examines the specific implementation details found in the OtterMind/Chat2DB repository, focusing on how the platform manages large file data import and export operations.

Streaming Architecture for Large File Processing

Export Batch Processing with DefaultDBManager

The export workflow centers on DefaultDBManager.java, where the exportTableData method orchestrates data extraction using a constant batch size approach. The system defines DEFAULT_EXPORT_BATCH_SIZE = 100000 (100,000 rows) to control memory allocation.

Rather than loading entire tables into memory, the implementation repeatedly calls fetchAllTableRecords with this batch size, passing a consumer that streams each chunk directly to the output stream. This pattern ensures that only one batch resides in heap memory at any moment, enabling exports of tables containing hundreds of millions of rows while maintaining predictable resource consumption.

Import Streaming via IImportStrategy

On the import side, ImportFactory.java manages format-specific strategies through the IImportStrategy interface. Concrete implementations—including CSVImporter, XLSXImporter, JSONImporter, and SQLImporter—extend BaseImporter or BaseExcelImporter, which parse source files in configurable chunks.

The import pipeline utilizes SyncSqlBatchHandler to execute generated INSERT statements in groups, preventing memory accumulation even when processing files exceeding several gigabytes. Each importer reads the input stream incrementally, transforming data and dispatching it to the database without materializing the entire source file.

Core Components and File Paths

The streaming system relies on specific components distributed across the Chat2DB codebase:

Batch Configuration and Memory Management

Chat2DB employs three critical design patterns to manage large file data import and export efficiently:

Result-set consumer pattern: DefaultSQLExecutor.fetchAllTableRecords accepts a functional consumer that processes each ResultSet chunk individually, enabling true row-by-row handling without materializing complete result sets in memory.

Configurable batch sizing: While exports default to 100,000 rows, the architecture supports client-side parameter overrides. Import batches derive sizing from uploaded file metadata or fixed limits defined within BaseImporter subclasses.

Async context isolation: ImportAsyncContext and analogous export contexts isolate streaming operations from the main execution thread. These contexts record progress messages incrementally, which the UI consumes via WebSocket or long-polling to display real-time status without blocking the transfer operation.

Task Coordination and Progress Tracking

TaskTransferServiceImpl wraps low-level import/export logic within the ITaskDataImportService and ITaskDataExportService interfaces. This high-level task API creates ImportAsyncContext instances (or corresponding export contexts) that coordinate the streaming process while logging execution progress.

The task service enables the UI to display accurate progress indicators for multi-gigabyte transfers, as the context updates status messages incrementally while rows flow through the batch handlers. This architecture ensures that users can monitor long-running operations without sacrificing system stability.

Practical Implementation Examples

Export a table to CSV via the REST API:

curl -X POST "http://localhost:10825/api/task/export/sqlFile" \
     -H "Content-Type: application/json" \
     -d '{
           "dataSourceId": 1,
           "databaseName": "demo",
           "schemaName": "public",
           "tableName": "orders",
           "exportType": "CSV",
           "exportPath": "/tmp/orders.csv",
           "scope": "TABLE"
         }'

Import a large CSV file using a Java client:

// Java client using Spring RestTemplate
RestTemplate rest = new RestTemplate();
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("dataSourceId", 1);
body.add("importType", "CSV");
body.add("importPath", new File("/tmp/large_orders.csv"));
HttpEntity<MultiValueMap<String, Object>> request = new HttpEntity<>(body);
ResponseEntity<Long> resp = rest.postForEntity(
        "http://localhost:10825/api/task/import/sqlFile",
        request,
        Long.class);
System.out.println("Import task id = " + resp.getBody());

Register a custom importer for new file formats:

public class MyFormatImporter extends BaseImporter implements IImportStrategy {
    @Override
    protected void doImportData(ImportAsyncContext ctx, List<TableColumn> columns) {
        // read the input stream in chunks, build INSERT statements, and execute them
    }
}

// In ImportFactory.java
private static final Map<String, IImportStrategy> exports = Map.of(
        "xls",   new XLSImporter(),
        "xlsx",  new XLSXImporter(),
        "csv",   new CSVImporter(),
        "json",  new JSONImporter(),
        "sql",   new SQLImporter(),
        "myfmt", new MyFormatImporter()   // ← newly added
);

Summary

  • Chat2DB uses a fixed batch size of 100,000 rows for exports via DefaultDBManager, defined as DEFAULT_EXPORT_BATCH_SIZE, preventing memory overflow when processing large tables.
  • Pluggable import strategies through IImportStrategy and ImportFactory support CSV, XLSX, JSON, and SQL formats with streaming parsers that extend BaseImporter.
  • Async contexts provide real-time progress tracking while keeping memory usage constant during multi-gigabyte transfers through ImportAsyncContext.
  • Result-set consumers enable row-by-row processing via fetchAllTableRecords without materializing entire datasets in RAM.

Frequently Asked Questions

What is the default batch size for Chat2DB exports?

The default batch size is 100,000 rows, defined as the constant DEFAULT_EXPORT_BATCH_SIZE in DefaultDBManager.java. This value controls how many rows fetchAllTableRecords retrieves and writes to the output stream in each iteration, ensuring predictable memory usage regardless of total table size.

How does Chat2DB prevent memory exhaustion when importing multi-gigabyte files?

Chat2DB employs streaming importers that extend BaseImporter or BaseExcelImporter, reading files in chunks rather than loading them entirely into memory. The SyncSqlBatchHandler executes generated INSERT statements in groups, while ImportAsyncContext isolates the operation from the main thread and reports progress incrementally to prevent heap overflow.

Can I add support for custom file formats in Chat2DB?

Yes, you can implement the IImportStrategy interface and extend BaseImporter to handle custom formats. Register your implementation in ImportFactory.java by adding it to the exports map with the appropriate file extension key, following the pattern used for CSVImporter and XLSXImporter.

Does Chat2DB support real-time progress tracking for large data transfers?

Yes, the system uses ImportAsyncContext for imports and corresponding async contexts for exports to record progress incrementally. The UI consumes these updates via WebSocket or long-polling, allowing users to monitor transfer status for files exceeding several gigabytes without blocking the interface or compromising system stability.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →