How TeslaMate's Import Module Migrates Data from TeslaFi

TeslaMate's TeslaFi import module converts exported CSV files into vehicle-state events through a fake API pipeline, parsing and normalizing historical data in lib/teslamate/import.ex and lib/teslamate/import/line_parser.ex to recreate vehicle records without requiring live Tesla API access.

The teslamate-org/teslamate repository provides a dedicated import pipeline that allows users to migrate historical driving and charging data from TeslaFi into TeslaMate's database. This module treats CSV exports as event streams, transforming them into native TeslaApi.Vehicle structs that the application processes as if they were real-time API responses.

Overview of the TeslaFi Import Architecture

The migration workflow relies on two core components working in concert. TeslaMate.Import orchestrates the state machine, directory scanning, and process supervision, while TeslaMate.Import.LineParser handles the granular transformation of CSV rows into structured vehicle data. Together, they create a fake API process that feeds historical data into TeslaMate's standard vehicle processing pipeline.

Step 1: CSV Discovery and File Validation

When TeslaMate.Import.run/1 initiates the import, the state machine transitions from :idle to process the :read_directory event (lines 100‑112 of lib/teslamate/import.ex). The module scans the directory specified by the IMPORT_DIR environment variable (defaulting to ./import) and identifies valid TeslaFi export files:

{:ok, files} = File.ls(path)
files
|> Enum.map(fn n -> %{date: parse_fname(n), path: Path.join([path, n])} end)
|> Enum.reject(fn %{date: date} -> is_nil(date) end)
|> Enum.sort_by(fn %{date: date} -> date end)

The parse_fname/1 function (lines 93‑104) recognizes two specific naming patterns:

• MMYYYY.csv (legacy format)
• TeslaFiMMYYYY.csv (current TeslaFi export format)

Files that do not match these patterns are silently ignored, ensuring only relevant CSV data enters the pipeline.

Step 2: Concurrent CSV Streaming and Parsing

Upon receiving the :import event (lines 120‑131), the module calls create_event_streams/2 to process each file. The implementation uses Elixir's Task.async_stream/4 to parse rows concurrently while maintaining order:

rows
|> Task.async_stream(&LineParser.parse(&1, tz), timeout: :infinity, ordered: true)
|> Stream.map(fn {:ok, vehicle} -> vehicle end)

Each CSV is streamed via File.stream!/2 and parsed with CSV.parse/1, then handed to LineParser.parse/2 (lines 238‑245) for transformation into vehicle structs.

Step 3: Data Normalization and Vehicle Struct Creation

The LineParser.parse/2 function (lines 18‑22) reduces each CSV row into a vehicle map by folding column values into a @default_vehicle base struct. The into_vehicle/3 function (lines 74‑133) handles specific data types:

• Timestamp conversion: Parses the Date column using Timex (lines 84‑88) and converts it to Unix milliseconds in the user-selected timezone, injecting this timestamp into all nested state maps (charge_state, climate_state, drive_state, vehicle_config, vehicle_state) at lines 108‑110.
• Boolean and numeric coercion: Helpers like map_value/2 and to_float/1 convert string representations of "TRUE", "FALSE", and numeric values into proper Elixir types.
• Vehicle ID fallback: When the CSV contains an empty vehicle_id field, the parser injects the value from the TESLAFI_IMPORT_VEHICLE_ID environment variable (lines 79‑81).
• Validation: If a row contains a VIN different from the initially detected vehicle, the parser raises a :vehicle_changed error (lines 51‑57) to prevent cross-contamination of data.

The assembled map is finally converted into a %TeslaApi.Vehicle{} struct via Vehicle.result/1.

Step 4: Fake API Simulation and Database Persistence

Before streaming begins, create_car/1 (lines 77‑99) iterates through the event streams to locate the first valid vehicle entry containing a vin, vehicle_id, and internal id. It then persists the car using Vehicles.create_or_update!/1 and attaches a minimal CarSettings struct (lines 91‑96) that disables streaming while keeping the car enabled.

The system then starts a fake API process (TeslaMate.Import.FakeApi, lines 40‑46) seeded with the prepared event streams. This process mimics the real Tesla API but yields imported historical data instead of live responses. A vehicle process (TeslaApi.Vehicle) is started with import?: true (lines 48‑54) and linked to the fake API, allowing the rest of TeslaMate to consume the historical data transparently.

Step 5: Import Completion and Cleanup

As each stream finishes processing, it sends a {:done, date} message to the state machine (lines 63‑68). When all streams report completion, the :done handler (lines 70‑80) executes:

• Terminates the vehicle process
• Normalizes the car's final state via Log.complete_current_state/1 and Log.create_current_state/1
• Broadcasts a :complete status via Phoenix PubSub to update the UI (lib/teslamate_web/live/import_live/index.html.heex)

If any stream encounters an error—such as detecting data for a different vehicle—the import aborts immediately with {:error, reason}, preventing partial or corrupted migrations.

Practical Usage Examples

Starting an Import via Console

Ensure CSV files are placed in the import directory, then execute:

# Set the import directory (optional)

# Files should be named like: ./import/TeslaFi062019.csv

{:ok, _pid} = TeslaMate.Import.start_link(name: :teslamate_import, directory: "./import")
:ok = TeslaMate.Import.run("Europe/Berlin")

Minimal TeslaFi CSV Format

Date,State,charge_state,drive_state,climate_state,vehicle_id,vin,latitude,longitude
2020-06-01 12:34:56,online,85,37.7749,-122.4194,38.5,1,5YJ3E1EA5KF....

Handling Environment Variables

export TESLAFI_IMPORT_VEHICLE_ID=42   # Default vehicle ID for empty CSV fields

export IMPORT_DIR=/path/to/csv/files
mix run -e "TeslaMate.Import.run(\"UTC\")"

Summary

• TeslaMate's import module migrates TeslaFi data by treating CSV exports as event streams consumed through a fake API interface.
• File discovery relies on specific naming patterns (TeslaFiMMYYYY.csv) scanned from IMPORT_DIR.
• Concurrent parsing uses Task.async_stream/4 and LineParser.parse/2 to transform rows into %TeslaApi.Vehicle{} structs with timezone-aware timestamps.
• Data integrity is enforced through VIN validation and the TESLAFI_IMPORT_VEHICLE_ID fallback for missing identifiers.
• Database integration occurs via Vehicles.create_or_update!/1, creating a car record that the standard TeslaMate pipeline populates with historical states.

Frequently Asked Questions

What file naming conventions does the TeslaFi import module require?

The import module recognizes two patterns in parse_fname/1: MMYYYY.csv for legacy exports and TeslaFiMMYYYY.csv for current TeslaFi formats. Files must use these exact patterns to be processed; all other filenames are ignored during the directory scan.

How does the module handle different timezones in historical data?

The LineParser.parse/2 function accepts a timezone parameter (e.g., "Europe/Berlin") and uses Timex to convert the CSV's Date column into Unix millisecond timestamps normalized to that timezone. This timestamp is then injected into all nested vehicle state maps to ensure consistent temporal alignment.

What happens if the CSV files contain data from multiple vehicles?

If LineParser detects a VIN mismatch between rows (lines 51‑57), it raises a :vehicle_changed error that aborts the entire import process. This prevents accidental merging of data from different vehicles into a single TeslaMate car record.

Can I import TeslaFi data without using the web interface?

Yes. The import can be initiated programmatically by calling TeslaMate.Import.run/1 with a timezone string from any Elixir context, such as a Mix task or IEx session, provided the IMPORT_DIR contains the properly named CSV files and the environment variables are configured.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}