# How DBX Implements Table Data Import from CSV and Excel Files: A Technical Deep Dive

> Learn how DBX handles CSV and Excel imports with its three-stage pipeline. Discover format detection, JSON parsing, and batched SQL inserts for efficient data loading.

- Repository: [skyler/dbx](https://github.com/t8y2/dbx)
- Tags: deep-dive
- Published: 2026-07-05

---

**DBX implements table data import from CSV and Excel files through a three-stage pipeline that detects source formats based on file extensions, parses files into a uniform JSON intermediate representation using the `csv` crate for delimited files and Calamine for Excel, then executes batched SQL INSERT statements with configurable streaming for CSV and in-memory processing for Excel.**

The DBX codebase (t8y2/dbx) handles **table data import from CSV and Excel files** by treating both formats as import sources that normalize into a `ParsedImportFile` structure. Located in [`crates/dbx-core/src/table_import.rs`](https://github.com/t8y2/dbx/blob/main/crates/dbx-core/src/table_import.rs), the implementation uses format-specific parsers—streaming for CSV/TSV and workbook-based extraction for Excel—before mapping columns and executing database inserts.

## Source Format Detection

Import processing begins with `import_file_kind()` (lines 1–16), which inspects file extensions to return a `TableImportSourceFormat` enum variant. The function `source_format_for_path()` maps extensions like `.csv`, `.tsv`, `.txt`, and `.xlsx` to their respective formats.

```rust
let format = effective_source_format(&request.file_path, request.source_format)?;

```

 callers invoke `effective_source_format()` to either use the user-supplied format or auto-detect from the path. This determines whether the pipeline uses the delimited parser or the Excel parser in subsequent stages.

## Parsing CSV and TSV Files

### Configuration and Delimited Options

For CSV and TSV files, DBX builds a parser configuration via `effective_delimited_config()` (lines 58–71). This function reads `TableImportParseOptions` to set delimiters (defaulting to `,` for CSV and `\t` for TSV), header flags, trimming behavior, and null-handling rules.

### Streaming Parser Implementation

The actual parsing happens in `parse_delimited_reader()` (lines 82–88), which constructs a `csv::Reader` from the configuration. The function extracts headers (or synthesizes `column_1`, `column_2`, etc. when headers are absent) and iterates over records, converting each cell to JSON using `csv_value_with_config()`. This operates in a **streaming fashion**, reading rows line-by-line to minimize memory usage.

The entry point `parse_delimited_bytes_with_options()` (lines 85–94) orchestrates this process and returns a `ParsedImportFile` containing `columns`, `rows`, and `total_rows`.

## Parsing Excel Workbooks

### Memory Constraints and Sheet Selection

Unlike CSV, Excel parsing is **non-streaming** and loads the entire workbook into memory. Before parsing, `ensure_non_streaming_file_size()` (lines 618–632) enforces a default 100 MiB size limit to prevent memory exhaustion.

The function `parse_xlsx_file_with_options()` (lines 441–610) uses the Calamine crate's `open_workbook_auto()` to load the file. It selects sheets based on `sheet_name`, `sheet_index`, or defaults to the first sheet (lines 447–558).

### Cell Conversion and Header Handling

Once a sheet is selected, the parser extracts headers if `has_header` is true; otherwise it generates synthetic column names (lines 568–580). Each Calamine `Data` cell converts to JSON via `xlsx_cell_value()` (lines 605–620). The entire sheet transforms into a `ParsedImportFile` before proceeding to SQL generation.

## Column Mapping and SQL Batch Generation

### Validating Column Mappings

Users supply `TableImportColumnMapping` objects to map source columns to target database columns. The function `mapping_indexes_for_columns()` (lines 776–796) validates these mappings, ensuring each source column exists and each target appears only once, returning index pairs for the transformation.

### Building INSERT Batches

**For CSV/TSV (streaming):** Rows accumulate up to `effective_batch_size`, though Oracle-compatible databases force single-row batches. `build_import_insert_batch_from_rows()` generates `INSERT … VALUES …` statements using `generate_insert_typed()` (lines 702–735), executing via `execute_on_pool()`.

**For Excel (non-streaming):** After the full file parses into memory, `build_import_insert_batches()` (lines 647–682) splits `ParsedImportFile.rows` into chunks and produces one SQL batch per chunk.

The central orchestrator `import_table_file_core()` (lines 911–1248) coordinates parsing, mapping, and execution while emitting `TableImportProgress` callbacks and handling errors through `emit_import_error()` (lines 693–720).

## Preview Mode

For UI previews, `preview_table_import_file_with_request()` (lines 758–896) runs the same parsing logic but stops after a configurable `preview_limit` (default 50 rows). It also returns available sheet names for Excel imports, allowing users to inspect data before committing to the full import.

## Implementation Examples

### Preview a CSV File

To inspect CSV data before importing:

```rust
use dbx_core::table_import::{TableImportPreviewRequest, preview_table_import_file_with_request};

let request = TableImportPreviewRequest {
    file_path: "data/users.csv".into(),
    source_format: None,               // auto-detect from extension
    parse_options: Default::default(),
    preview_limit: Some(20),
    ..Default::default()
};

let preview = preview_table_import_file_with_request(request).await?;
println!("Columns: {:?}", preview.columns);
println!("First rows: {:?}", preview.rows);

```

*Relevant source:* `preview_table_import_file_with_request` (lines 758–896) in [`crates/dbx-core/src/table_import.rs`](https://github.com/t8y2/dbx/blob/main/crates/dbx-core/src/table_import.rs).

### Import an Excel Workbook

To import a specific sheet with column mappings:

```rust
use dbx_core::table_import::{
    TableImportRequest, TableImportColumnMapping, TableImportMode,
    import_table_file_core,
};

let req = TableImportRequest {
    import_id: "imp-123".into(),
    connection_id: "conn-1".into(),
    database: "demo".into(),
    schema: "public".into(),
    table: "customers".into(),
    file_path: "/tmp/customers.xlsx".into(),
    source_format: Some(TableImportSourceFormat::Excel),
    parse_options: TableImportParseOptions {
        sheet_name: Some("2024".into()),
        ..Default::default()
    },
    mappings: vec![
        TableImportColumnMapping { source_column: "id".into(), target_column: "customer_id".into() },
        TableImportColumnMapping { source_column: "name".into(), target_column: "full_name".into() },
    ],
    mode: TableImportMode::Append,
    batch_size: 500,
};

let summary = import_table_file_core(&state, &req, &DatabaseType::Postgres, "pool-1", |_| async { false }, |_| {}).await?;
println!("Imported {} rows", summary.rows_imported);

```

*Relevant source:* `import_table_file_core` (lines 911–1248) in [`crates/dbx-core/src/table_import.rs`](https://github.com/t8y2/dbx/blob/main/crates/dbx-core/src/table_import.rs).

## Summary

- **Format Detection:** DBX uses `import_file_kind()` and `source_format_for_path()` to identify CSV, TSV, and Excel files by extension.
- **CSV Processing:** Streaming parser using the `csv` crate with configurable delimiters and headers via `parse_delimited_reader()`.
- **Excel Processing:** In-memory parsing using Calamine's `open_workbook_auto()` with a 100 MiB size limit and sheet selection logic.
- **Column Mapping:** Validation and index mapping through `mapping_indexes_for_columns()` before SQL generation.
- **Batch Execution:** Streaming batches for CSV (with Oracle-compatible single-row fallback) and chunk-based batches for Excel via `build_import_insert_batches()`.
- **Preview Support:** `preview_table_import_file_with_request()` allows inspection of 50 rows (configurable) without importing.

## Frequently Asked Questions

### What is the maximum file size for Excel imports in DBX?

DBX enforces a default 100 MiB limit for Excel files via `ensure_non_streaming_file_size()` (lines 618–632) because the Calamine-based parser loads the entire workbook into memory. This prevents memory exhaustion during the import process.

### Does DBX stream CSV files during import or load them entirely into memory?

DBX streams CSV and TSV files using `parse_delimited_reader()`, which processes rows line-by-line through the `csv` crate. This streaming approach minimizes memory usage and allows efficient processing of large delimited files, contrasting with the in-memory Excel parser.

### How does DBX handle column mapping between source files and database tables?

DBX accepts `TableImportColumnMapping` objects that define source-to-target column relationships. The function `mapping_indexes_for_columns()` (lines 776–796) validates these mappings, ensuring source columns exist and targets are unique, then returns index pairs used during SQL batch generation.

### Which Rust crates does DBX use for parsing CSV and Excel files?

DBX uses the `csv` crate for parsing delimited text files (CSV, TSV) with `csv::ReaderBuilder`, and the Calamine crate for Excel workbooks via `open_workbook_auto()`. These implementations reside in [`crates/dbx-core/src/table_import.rs`](https://github.com/t8y2/dbx/blob/main/crates/dbx-core/src/table_import.rs).