How TeslaMate's HTTP Client Implements Circuit Breakers Using the Erlang Fuse Library

TeslaMate wraps its Tesla API HTTP client in circuit breakers using the Erlang :fuse library to halt requests after consecutive failures, isolating the global API service and individual vehicle processes from cascading errors.

TeslaMate is an open-source, self-hosted data logger for Tesla vehicles that requires robust resilience against Tesla API rate limits and intermittent outages. By implementing circuit breaker logic in lib/teslamate/api.ex and lib/teslamate/vehicles/vehicle.ex, the application automatically detects fault conditions and stops sending requests until the service recovers.

Installing Global and Per-Vehicle Circuit Breakers

The circuit breaker implementation relies on named fuses created during process initialization. Each fuse uses a standard strategy that defines a failure threshold and a time window, paired with a reset strategy that determines how long the circuit stays open.

Global API Client Fuse

When the TeslaMate.Api GenServer starts, it installs a fuse keyed to the process name that protects all unauthorized error scenarios:

# lib/teslamate/api.ex – init/1 (lines 101-105)

:fuse.install(
  fuse_name(name),
  {{:standard, 5, :timer.minutes(10)}, {:reset, :timer.hours(9999)}}
)

• fuse_name/1 generates an atom like :"#{name}.unauthorized" (e.g., :tesla_api.unauthorized).
• The policy {:standard, 5, :timer.minutes(10)} melts the fuse after 5 failures within 10 minutes.
• The reset strategy {:reset, :timer.hours(9999)} keeps the circuit open for 9,999 hours after it blows, effectively requiring manual intervention or a successful reset to close.

Per-Vehicle Fuses for Granular Protection

Each vehicle process in TeslaMate.Vehicles.Vehicle installs two dedicated fuses during its init/1 callback:

# lib/teslamate/vehicles/vehicle.ex – init/1 (lines 198-207)

fuses = [
  {:vehicle_not_found, {{:standard, 8, :timer.minutes(20)}, {:reset, :timer.minutes(10)}}},
  {:api_error,        {{:standard, 3, :timer.minutes(10)}, {:reset, :timer.minutes(5)}}}
]

for {key, opts} <- fuses do
  name = fuse_name(key, data.car.id)
  :ok = :fuse.install(name, opts)
  :ok = :fuse.circuit_enable(name)
end

• api_error blows after 3 failures in 10 minutes and resets after 5 minutes.
• vehicle_not_found tolerates 8 failures over 20 minutes before blowing, reflecting its lower severity, and resets after 10 minutes.
• :fuse.circuit_enable/1 activates the fuse so it begins monitoring failures immediately.

Tripping the Circuit on Authentication Failures

When an API call returns an unauthorized error, the client melts the fuse to record the failure, then checks the fuse state to decide whether to retry or abort.

Melting the Fuse in handle_result

The handle_result/3 function in lib/teslamate/api.ex processes the Tesla API response. On an unauthorized error, it melts the fuse and interrogates the circuit state:

# lib/teslamate/api.ex – handle_result/3 (lines 72-84)

{:error, %TeslaApi.Error{reason: :unauthorized}} ->
  :ok = :fuse.melt(fuse_name(name))

  case :fuse.ask(fuse_name(name), :sync) do
    :blown ->
      true = :ets.delete(name, :auth)
      {:error, :not_signed_in}
    :ok ->
      send(name, :refresh_auth)
      {:error, :unauthorized}
  end

• :fuse.melt/1 increments the failure counter for the named circuit.
• :fuse.ask/2 with the :sync option returns :blown if the threshold was exceeded, or :ok if the circuit is still closed.
• If the fuse is blown, the client deletes the cached authentication from ETS and returns {:error, :not_signed_in}, forcing a fresh sign-in.
• If the circuit remains closed, it triggers an asynchronous token refresh via send(name, :refresh_auth).

Querying Circuit State Before Requests

Before performing health checks or state-changing operations, vehicle processes validate that their specific fuses are intact.

Checking Health with healthy?/1

The healthy?/1 function queries both the api_error and vehicle_not_found fuses to determine if the vehicle process should attempt API calls:

# lib/teslamate/vehicles/vehicle.ex – healthy?/1 (lines 49-55)

def healthy?(car_id) do
  with :ok <- :fuse.ask(fuse_name(:api_error, car_id), :sync),
       :ok <- :fuse.ask(fuse_name(:vehicle_not_found, car_id), :sync) do
    true
  else
    :blown -> false
  end
end

If either fuse returns :blown, the function returns false, and the vehicle process skips polling until the reset interval expires.

Resetting Fuses After Successful Recovery

When the API client successfully authenticates or completes a request after failures, it clears the fuse state to restore normal operation.

Resetting the Global Fuse

After a successful sign-in, the global unauthorized fuse is explicitly reset:

# lib/teslamate/api.ex – after successful sign-in (line 153)

:ok = :fuse.reset(fuse_name(state.name))

:fuse.reset/1 immediately closes the circuit and clears the failure counter, allowing subsequent requests to proceed without waiting for the automatic reset timer.

Practical Code Examples

Starting the API Client with Circuit Breaker Protection

When you start the API client, the fuse installs automatically with a 5-failure threshold:

{:ok, pid} = TeslaMate.Api.start_link(name: :tesla_api)

# Fuse :tesla_api.unauthorized is now active

Checking Vehicle Health Before Polling

Verify a vehicle's circuit state before making API requests:

car_id = 42

if TeslaMate.Vehicles.Vehicle.healthy?(car_id) do
  {:ok, state} = TeslaMate.Vehicles.Vehicle.get_state(car_id)
else
  IO.puts("Circuit breaker open for vehicle #{car_id} – skipping request")
end

Manually Resetting a Blown Fuse

After manually refreshing a token or resolving an API issue, reset the fuse to resume operations immediately:

:fuse.reset(:tesla_api.unauthorized)

Summary

• TeslaMate.Api installs a global fuse in lib/teslamate/api.ex that blows after 5 unauthorized errors in 10 minutes.
• TeslaMate.Vehicles.Vehicle creates per-vehicle fuses for api_error (3 failures) and vehicle_not_found (8 failures) to isolate individual vehicle issues.
• :fuse.melt/1 records failures, while :fuse.ask/2 checks if the circuit is :blown or :ok before proceeding.
• :fuse.reset/1 manually clears the failure state after successful recovery, whereas automatic reset occurs after the configured timeout.

Frequently Asked Questions

What triggers the circuit breaker to open in TeslaMate?

The circuit breaker opens when :fuse.melt/1 records enough consecutive failures to exceed the threshold defined in the fuse policy. For the global API client, this occurs after 5 unauthorized errors within 10 minutes. For individual vehicles, the api_error fuse blows after 3 failures in 10 minutes.

How does TeslaMate recover after a circuit breaker trips?

Recovery happens automatically after the reset timer expires (5 minutes for api_error, 10 minutes for vehicle_not_found, or 9,999 hours for the global unauthorized fuse). Alternatively, calling :fuse.reset/1 manually after a successful authentication immediately closes the circuit and clears the failure count.

Why does TeslaMate use per-vehicle fuses instead of only a global fuse?

Per-vehicle fuses in lib/teslamate/vehicles/vehicle.ex provide finer-grained fault isolation. If one vehicle consistently returns "not found" or API errors, it can be isolated without blocking requests for other vehicles sharing the same Tesla account. This prevents a single problematic vehicle from degrading the entire fleet's data syncing.

What is the difference between :fuse.melt and :fuse.reset?

:fuse.melt/1 increments the failure counter and potentially blows the circuit if the threshold is exceeded. :fuse.reset/1 clears all failure history and immediately closes an open circuit, typically called after a successful request or manual recovery to restore service before the automatic reset timer expires.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}