# How Chat2DB Handles Large File Data Import and Export: Streaming Architecture Explained > Discover how Chat2DB efficiently handles large file data import and export with its streaming architecture, ensuring constant memory usage for massive datasets. - Repository: [OtterMind/Chat2DB](https://github.com/OtterMind/Chat2DB) - Tags: architecture - Published: 2026-07-26 --- **Chat2DB processes massive datasets by streaming rows in configurable batches of 100,000 records, using asynchronous contexts and pluggable import strategies to maintain constant memory usage regardless of file size.** Chat2DB, an open-source database management tool maintained by OtterMind, implements a sophisticated streaming architecture to handle multi-gigabyte data transfers without exhausting system memory. Unlike conventional tools that load entire files into RAM, Chat2DB processes data incrementally through batch handlers and result-set consumers. This article examines the specific implementation details found in the OtterMind/Chat2DB repository, focusing on how the platform manages large file data import and export operations. ## Streaming Architecture for Large File Processing ### Export Batch Processing with DefaultDBManager The export workflow centers on [`DefaultDBManager.java`](https://github.com/OtterMind/Chat2DB/blob/main/DefaultDBManager.java), where the `exportTableData` method orchestrates data extraction using a **constant batch size approach**. The system defines `DEFAULT_EXPORT_BATCH_SIZE = 100000` (100,000 rows) to control memory allocation. Rather than loading entire tables into memory, the implementation repeatedly calls `fetchAllTableRecords` with this batch size, passing a consumer that streams each chunk directly to the output stream. This pattern ensures that only one batch resides in heap memory at any moment, enabling exports of tables containing hundreds of millions of rows while maintaining predictable resource consumption. ### Import Streaming via IImportStrategy On the import side, [`ImportFactory.java`](https://github.com/OtterMind/Chat2DB/blob/main/ImportFactory.java) manages format-specific strategies through the `IImportStrategy` interface. Concrete implementations—including `CSVImporter`, `XLSXImporter`, `JSONImporter`, and `SQLImporter`—extend `BaseImporter` or `BaseExcelImporter`, which parse source files in configurable chunks. The import pipeline utilizes `SyncSqlBatchHandler` to execute generated `INSERT` statements in groups, preventing memory accumulation even when processing files exceeding several gigabytes. Each importer reads the input stream incrementally, transforming data and dispatching it to the database without materializing the entire source file. ## Core Components and File Paths The streaming system relies on specific components distributed across the Chat2DB codebase: - [`chat2db-community-server/chat2db-community-spi/src/main/java/ai/chat2db/spi/DefaultDBManager.java`](https://github.com/OtterMind/Chat2DB/blob/main/chat2db-community-server/chat2db-community-spi/src/main/java/ai/chat2db/spi/DefaultDBManager.java): Contains the core export logic, including `DEFAULT_EXPORT_BATCH_SIZE` and the `exportTableData` method that implements batch-wise result set consumption. - [`chat2db-community-server/chat2db-community-web/src/main/java/ai/chat2db/community/web/api/adapter/db/DmlExportDeliveryAdapter.java`](https://github.com/OtterMind/Chat2DB/blob/main/chat2db-community-server/chat2db-community-web/src/main/java/ai/chat2db/community/web/api/adapter/db/DmlExportDeliveryAdapter.java): Resolves output streams for different deployment modes, opening HTTP responses for server installations and temporary files for desktop clients via `DownloadUtil.createDownloadFile`. - [`chat2db-community-server/chat2db-community-domain/chat2db-community-domain-core/src/main/java/ai/chat2db/community/domain/core/impl/task/imports/ImportFactory.java`](https://github.com/OtterMind/Chat2DB/blob/main/chat2db-community-server/chat2db-community-domain/chat2db-community-domain-core/src/main/java/ai/chat2db/community/domain/core/impl/task/imports/ImportFactory.java): Maintains the `exports` map that associates file extensions with concrete `IImportStrategy` implementations. - [`chat2db-community-server/chat2db-community-web/src/main/java/ai/chat2db/community/web/api/controller/TaskExportController.java`](https://github.com/OtterMind/Chat2DB/blob/main/chat2db-community-server/chat2db-community-web/src/main/java/ai/chat2db/community/web/api/controller/TaskExportController.java): Exposes the REST endpoint `/api/task/export/sqlFile` that initiates export tasks. - [`chat2db-community-server/chat2db-community-web/src/main/java/ai/chat2db/community/web/api/controller/TaskImportController.java`](https://github.com/OtterMind/Chat2DB/blob/main/chat2db-community-server/chat2db-community-web/src/main/java/ai/chat2db/community/web/api/controller/TaskImportController.java): Exposes the REST endpoint `/api/task/import/sqlFile` that triggers import operations. ## Batch Configuration and Memory Management Chat2DB employs three critical design patterns to manage large file data import and export efficiently: **Result-set consumer pattern**: `DefaultSQLExecutor.fetchAllTableRecords` accepts a functional consumer that processes each `ResultSet` chunk individually, enabling true row-by-row handling without materializing complete result sets in memory. **Configurable batch sizing**: While exports default to 100,000 rows, the architecture supports client-side parameter overrides. Import batches derive sizing from uploaded file metadata or fixed limits defined within `BaseImporter` subclasses. **Async context isolation**: `ImportAsyncContext` and analogous export contexts isolate streaming operations from the main execution thread. These contexts record progress messages incrementally, which the UI consumes via WebSocket or long-polling to display real-time status without blocking the transfer operation. ## Task Coordination and Progress Tracking `TaskTransferServiceImpl` wraps low-level import/export logic within the `ITaskDataImportService` and `ITaskDataExportService` interfaces. This high-level task API creates `ImportAsyncContext` instances (or corresponding export contexts) that coordinate the streaming process while logging execution progress. The task service enables the UI to display accurate progress indicators for multi-gigabyte transfers, as the context updates status messages incrementally while rows flow through the batch handlers. This architecture ensures that users can monitor long-running operations without sacrificing system stability. ## Practical Implementation Examples Export a table to CSV via the REST API: ```bash curl -X POST "http://localhost:10825/api/task/export/sqlFile" \ -H "Content-Type: application/json" \ -d '{ "dataSourceId": 1, "databaseName": "demo", "schemaName": "public", "tableName": "orders", "exportType": "CSV", "exportPath": "/tmp/orders.csv", "scope": "TABLE" }' ``` Import a large CSV file using a Java client: ```java // Java client using Spring RestTemplate RestTemplate rest = new RestTemplate(); MultiValueMap body = new LinkedMultiValueMap<>(); body.add("dataSourceId", 1); body.add("importType", "CSV"); body.add("importPath", new File("/tmp/large_orders.csv")); HttpEntity> request = new HttpEntity<>(body); ResponseEntity resp = rest.postForEntity( "http://localhost:10825/api/task/import/sqlFile", request, Long.class); System.out.println("Import task id = " + resp.getBody()); ``` Register a custom importer for new file formats: ```java public class MyFormatImporter extends BaseImporter implements IImportStrategy { @Override protected void doImportData(ImportAsyncContext ctx, List columns) { // read the input stream in chunks, build INSERT statements, and execute them } } // In ImportFactory.java private static final Map exports = Map.of( "xls", new XLSImporter(), "xlsx", new XLSXImporter(), "csv", new CSVImporter(), "json", new JSONImporter(), "sql", new SQLImporter(), "myfmt", new MyFormatImporter() // ← newly added ); ``` ## Summary - **Chat2DB uses a fixed batch size of 100,000 rows** for exports via `DefaultDBManager`, defined as `DEFAULT_EXPORT_BATCH_SIZE`, preventing memory overflow when processing large tables. - **Pluggable import strategies** through `IImportStrategy` and `ImportFactory` support CSV, XLSX, JSON, and SQL formats with streaming parsers that extend `BaseImporter`. - **Async contexts** provide real-time progress tracking while keeping memory usage constant during multi-gigabyte transfers through `ImportAsyncContext`. - **Result-set consumers** enable row-by-row processing via `fetchAllTableRecords` without materializing entire datasets in RAM. ## Frequently Asked Questions ### What is the default batch size for Chat2DB exports? The default batch size is **100,000 rows**, defined as the constant `DEFAULT_EXPORT_BATCH_SIZE` in [`DefaultDBManager.java`](https://github.com/OtterMind/Chat2DB/blob/main/DefaultDBManager.java). This value controls how many rows `fetchAllTableRecords` retrieves and writes to the output stream in each iteration, ensuring predictable memory usage regardless of total table size. ### How does Chat2DB prevent memory exhaustion when importing multi-gigabyte files? Chat2DB employs **streaming importers** that extend `BaseImporter` or `BaseExcelImporter`, reading files in chunks rather than loading them entirely into memory. The `SyncSqlBatchHandler` executes generated `INSERT` statements in groups, while `ImportAsyncContext` isolates the operation from the main thread and reports progress incrementally to prevent heap overflow. ### Can I add support for custom file formats in Chat2DB? Yes, you can implement the `IImportStrategy` interface and extend `BaseImporter` to handle custom formats. Register your implementation in [`ImportFactory.java`](https://github.com/OtterMind/Chat2DB/blob/main/ImportFactory.java) by adding it to the `exports` map with the appropriate file extension key, following the pattern used for `CSVImporter` and `XLSXImporter`. ### Does Chat2DB support real-time progress tracking for large data transfers? Yes, the system uses `ImportAsyncContext` for imports and corresponding async contexts for exports to record progress incrementally. The UI consumes these updates via WebSocket or long-polling, allowing users to monitor transfer status for files exceeding several gigabytes without blocking the interface or compromising system stability.