# How TeslaMate Parses and Migrates CSV Data from TeslaFi and tesla-apiscraper

> Learn how TeslaMate imports TeslaFi CSV data with GenStateMachine. Stream files, parse rows, and migrate data effortlessly for a complete vehicle history.

- Repository: [TeslaMate/teslamate](https://github.com/teslamate-org/teslamate)
- Tags: how-to-guide
- Published: 2026-06-23

---

**TeslaMate imports historical vehicle telemetry from TeslaFi CSV exports (and converted tesla-apiscraper data) using a GenStateMachine pipeline that streams files chronologically, parses rows into API-compatible structs, and persists them through a fake API process that emulates live vehicle responses.**

The `teslamate-org/teslamate` repository provides a robust **CSV import from TeslaFi** system that migrates legacy driving data into its PostgreSQL schema. This implementation handles file discovery, timezone-normalized timestamp parsing, and parallel event processing to reconstruct complete vehicle histories including drives, charges, and positions.

## File Discovery and Pattern Matching

The import process begins in [`lib/teslamate/import.ex`](https://github.com/teslamate-org/teslamate/blob/main/lib/teslamate/import.ex) where the `TeslaMate.Import` GenStateMachine scans the default `import/` directory for CSV files. The system recognizes two filename patterns: `mmYYYY.csv` and `TeslaFimmYYYY.csv`.

The `parse_fname/1` function extracts the month and year from matching filenames to build a chronologically sorted list of files. This ensures data is processed in temporal order, preventing timeline inconsistencies during migration.

## Streaming CSV Parsing with NimbleCSV

Once files are identified, the system opens each as a lazy stream via `File.stream!` and passes it to `TeslaMate.Import.CSV.parse/1` in [`lib/teslamate/import/csv.ex`](https://github.com/teslamate-org/teslamate/blob/main/lib/teslamate/import/csv.ex). This module uses **NimbleCSV** to validate delimiters and skip empty files or malformed data.

The parser returns `{:error, :unsupported_delimiter}` or `{:error, :no_contents}` for invalid inputs, and yields an enumerable of maps (header → value) for valid files. This streaming approach keeps memory usage constant even when processing gigabytes of historical telemetry.

## Row-to-Struct Conversion and Timestamp Normalization

Each CSV row map undergoes transformation by `TeslaMate.Import.LineParser.parse/2` in [`lib/teslamate/import/line_parser.ex`](https://github.com/teslamate-org/teslamate/blob/main/lib/teslamate/import/line_parser.ex). This function:

- Converts string booleans like `"TRUE"`/`"FALSE"` to Elixir booleans
- Parses numeric strings into integers or floats
- Replaces empty strings or `"None"` values with `nil`
- Injects a UTC timestamp into nested state maps using **Timex** and the user-provided timezone

The parser places the normalized timestamp under the `"timestamp"` key within `charge_state`, `drive_state`, `climate_state`, and other sub-maps. Finally, `TeslaApi.Vehicle.result/1` constructs a complete `%TeslaApi.Vehicle{}` struct that mirrors live API responses.

## Parallel Processing and Fake API Emulation

The `create_event_streams/2` function wraps the parsing logic in `Task.async_stream/3` to process rows in parallel. The resulting stream of vehicle structs feeds into `TeslaMate.Import.FakeApi`, a lightweight process that emulates the official Tesla API.

A special import-mode Vehicle process (`import_<car_name>`) starts and links to this fake API, receiving events exactly as it would from live telemetry. This architecture allows TeslaMate to reuse its existing persistence pipelines without separate import-specific logic.

## Database Persistence and Post-Import Repair

As the fake API pushes events, the standard **Log** module functions handle persistence:

- `Log.create_current_state/1` inserts new position and state records
- `Log.complete_current_state/1` finalizes drive and charge sessions
- PostgreSQL tables `positions`, `drives`, and `charges` populate with the migrated data

When streams exhaust, the import stops the vehicle and fake API processes, then triggers `Repair.trigger_run/0` to recompute derived statistics and ensure data consistency across the migrated history.

## Handling tesla-apiscraper Conversions

For **tesla-apiscraper** users, TeslaMate expects data to be pre-converted to the TeslaFi CSV format. External tools read the apiscraper InfluxDB exports, normalize known data quirks, and output `TeslaFiMMYYYY.csv` files. Once placed in the `import/` directory, these files process through the identical pipeline described above, requiring no additional code within TeslaMate itself.

## Implementation Examples

### Starting an Import from IEx

```elixir

# Ensure CSV files exist in the import directory

{:ok, _pid} = TeslaMate.Import.start_link(directory: "/opt/app/import")

# Execute import with the data's source timezone

:ok = TeslaMate.Import.run("Europe/Berlin")

# Check current status

status = TeslaMate.Import.get_status()

```

### Parsed Vehicle Struct Format

```elixir
%TeslaApi.Vehicle{
  id: 12345,
  vehicle_id: 67890,
  vin: "5YJ3E1EA7KF123456",
  state: "online",
  charge_state: %{
    battery_level: 85,
    charging_state: "Complete",
    timestamp: 1_638_720_000_000
  },
  drive_state: %{
    latitude: 52.5200,
    longitude: 13.4050,
    timestamp: 1_638_719_500_000
  }
}

```

### Creating Event Streams Manually

```elixir
streams =
  TeslaMate.Import.create_event_streams(%TeslaMate.Import{
    files: [%{date: [2020, 5], path: "/opt/app/import/TeslaFi052020.csv"}],
    timezone: "Europe/Berlin"
  })

# Returns: [{[2020, 5], #Stream<...>}]

```

## Summary

- **File Discovery**: Scans `import/` directory for `mmYYYY.csv` or `TeslaFimmYYYY.csv` patterns using `parse_fname/1`
- **Streaming Parse**: Uses `TeslaMate.Import.CSV` with NimbleCSV to handle large files without memory bloat
- **Struct Conversion**: `LineParser.parse/2` normalizes types and timestamps, producing `%TeslaApi.Vehicle{}` structs
- **API Emulation**: `FakeApi` process feeds parsed data to import-mode Vehicle processes via standard event streams
- **Persistence**: Reuses existing `Log` functions to populate PostgreSQL with positions, drives, and charges
- **Repair**: Post-import repair run recalculates derived data for consistency

## Frequently Asked Questions

### What filename pattern does TeslaMate require for CSV imports?

TeslaMate recognizes files matching `mmYYYY.csv` or `TeslaFimmYYYY.csv` (e.g., `TeslaFi052020.csv`). The `parse_fname/1` function in [`lib/teslamate/import.ex`](https://github.com/teslamate-org/teslamate/blob/main/lib/teslamate/import.ex) extracts the month and year from these patterns to sort files chronologically before processing.

### How does TeslaMate handle timezone conversion during import?

The import accepts a timezone parameter (e.g., `"Europe/Berlin"` via `TeslaMate.Import.run/1`) that `LineParser.parse/2` uses with **Timex** to convert the CSV's naive "Date" column into UTC timestamps. These timestamps are injected into each state sub-map under the `"timestamp"` key.

### Can I import data from tesla-apiscraper directly?

No, TeslaMate requires pre-conversion to the TeslaFi CSV format. The tesla-apiscraper data must be exported from InfluxDB and transformed using external tools that output `TeslaFiMMYYYY.csv` files, which are then processed by the standard `TeslaMate.Import` pipeline.

### Will importing CSV data overwrite existing TeslaMate records?

The import uses a `date_limit` filter in `FakeApi` to emit only events older than your first existing TeslaMate state. The `Vehicles.create_or_update!/1` function creates or updates the car record, but historical events are inserted alongside existing data without overwriting newer records.