How TeslaMate Parses and Migrates CSV Data from TeslaFi and tesla-apiscraper

TeslaMate imports historical vehicle telemetry from TeslaFi CSV exports (and converted tesla-apiscraper data) using a GenStateMachine pipeline that streams files chronologically, parses rows into API-compatible structs, and persists them through a fake API process that emulates live vehicle responses.

The teslamate-org/teslamate repository provides a robust CSV import from TeslaFi system that migrates legacy driving data into its PostgreSQL schema. This implementation handles file discovery, timezone-normalized timestamp parsing, and parallel event processing to reconstruct complete vehicle histories including drives, charges, and positions.

File Discovery and Pattern Matching

The import process begins in lib/teslamate/import.ex where the TeslaMate.Import GenStateMachine scans the default import/ directory for CSV files. The system recognizes two filename patterns: mmYYYY.csv and TeslaFimmYYYY.csv.

The parse_fname/1 function extracts the month and year from matching filenames to build a chronologically sorted list of files. This ensures data is processed in temporal order, preventing timeline inconsistencies during migration.

Streaming CSV Parsing with NimbleCSV

Once files are identified, the system opens each as a lazy stream via File.stream! and passes it to TeslaMate.Import.CSV.parse/1 in lib/teslamate/import/csv.ex. This module uses NimbleCSV to validate delimiters and skip empty files or malformed data.

The parser returns {:error, :unsupported_delimiter} or {:error, :no_contents} for invalid inputs, and yields an enumerable of maps (header → value) for valid files. This streaming approach keeps memory usage constant even when processing gigabytes of historical telemetry.

Row-to-Struct Conversion and Timestamp Normalization

Each CSV row map undergoes transformation by TeslaMate.Import.LineParser.parse/2 in lib/teslamate/import/line_parser.ex. This function:

• Converts string booleans like "TRUE"/"FALSE" to Elixir booleans
• Parses numeric strings into integers or floats
• Replaces empty strings or "None" values with nil
• Injects a UTC timestamp into nested state maps using Timex and the user-provided timezone

The parser places the normalized timestamp under the "timestamp" key within charge_state, drive_state, climate_state, and other sub-maps. Finally, TeslaApi.Vehicle.result/1 constructs a complete %TeslaApi.Vehicle{} struct that mirrors live API responses.

Parallel Processing and Fake API Emulation

The create_event_streams/2 function wraps the parsing logic in Task.async_stream/3 to process rows in parallel. The resulting stream of vehicle structs feeds into TeslaMate.Import.FakeApi, a lightweight process that emulates the official Tesla API.

A special import-mode Vehicle process (import_<car_name>) starts and links to this fake API, receiving events exactly as it would from live telemetry. This architecture allows TeslaMate to reuse its existing persistence pipelines without separate import-specific logic.

Database Persistence and Post-Import Repair

As the fake API pushes events, the standard Log module functions handle persistence:

• Log.create_current_state/1 inserts new position and state records
• Log.complete_current_state/1 finalizes drive and charge sessions
• PostgreSQL tables positions, drives, and charges populate with the migrated data

When streams exhaust, the import stops the vehicle and fake API processes, then triggers Repair.trigger_run/0 to recompute derived statistics and ensure data consistency across the migrated history.

Handling tesla-apiscraper Conversions

For tesla-apiscraper users, TeslaMate expects data to be pre-converted to the TeslaFi CSV format. External tools read the apiscraper InfluxDB exports, normalize known data quirks, and output TeslaFiMMYYYY.csv files. Once placed in the import/ directory, these files process through the identical pipeline described above, requiring no additional code within TeslaMate itself.

Implementation Examples

Starting an Import from IEx

# Ensure CSV files exist in the import directory

{:ok, _pid} = TeslaMate.Import.start_link(directory: "/opt/app/import")

# Execute import with the data's source timezone

:ok = TeslaMate.Import.run("Europe/Berlin")

# Check current status

status = TeslaMate.Import.get_status()

Parsed Vehicle Struct Format

%TeslaApi.Vehicle{
  id: 12345,
  vehicle_id: 67890,
  vin: "5YJ3E1EA7KF123456",
  state: "online",
  charge_state: %{
    battery_level: 85,
    charging_state: "Complete",
    timestamp: 1_638_720_000_000
  },
  drive_state: %{
    latitude: 52.5200,
    longitude: 13.4050,
    timestamp: 1_638_719_500_000
  }
}

Creating Event Streams Manually

streams =
  TeslaMate.Import.create_event_streams(%TeslaMate.Import{
    files: [%{date: [2020, 5], path: "/opt/app/import/TeslaFi052020.csv"}],
    timezone: "Europe/Berlin"
  })

# Returns: [{[2020, 5], #Stream<...>}]

Summary

• File Discovery: Scans import/ directory for mmYYYY.csv or TeslaFimmYYYY.csv patterns using parse_fname/1
• Streaming Parse: Uses TeslaMate.Import.CSV with NimbleCSV to handle large files without memory bloat
• Struct Conversion: LineParser.parse/2 normalizes types and timestamps, producing %TeslaApi.Vehicle{} structs
• API Emulation: FakeApi process feeds parsed data to import-mode Vehicle processes via standard event streams
• Persistence: Reuses existing Log functions to populate PostgreSQL with positions, drives, and charges
• Repair: Post-import repair run recalculates derived data for consistency

Frequently Asked Questions

What filename pattern does TeslaMate require for CSV imports?

TeslaMate recognizes files matching mmYYYY.csv or TeslaFimmYYYY.csv (e.g., TeslaFi052020.csv). The parse_fname/1 function in lib/teslamate/import.ex extracts the month and year from these patterns to sort files chronologically before processing.

How does TeslaMate handle timezone conversion during import?

The import accepts a timezone parameter (e.g., "Europe/Berlin" via TeslaMate.Import.run/1) that LineParser.parse/2 uses with Timex to convert the CSV's naive "Date" column into UTC timestamps. These timestamps are injected into each state sub-map under the "timestamp" key.

Can I import data from tesla-apiscraper directly?

No, TeslaMate requires pre-conversion to the TeslaFi CSV format. The tesla-apiscraper data must be exported from InfluxDB and transformed using external tools that output TeslaFiMMYYYY.csv files, which are then processed by the standard TeslaMate.Import pipeline.

Will importing CSV data overwrite existing TeslaMate records?

The import uses a date_limit filter in FakeApi to emit only events older than your first existing TeslaMate state. The Vehicles.create_or_update!/1 function creates or updates the car record, but historical events are inserted alongside existing data without overwriting newer records.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}