# Provider Error Handling and Automatic Retry Mechanisms in Music Assistant

> Discover Music Assistant's robust error handling and automatic retry mechanisms. Learn how Music Assistant ensures reliable API requests with exponential backoff and rate-limiting for all music providers.

- Repository: [Music Assistant/server](https://github.com/music-assistant/server)
- Tags: deep-dive
- Published: 2026-06-16

---

**Music Assistant implements a centralized throttling and retry framework in [`music_assistant/helpers/throttle_retry.py`](https://github.com/music-assistant/server/blob/main/music_assistant/helpers/throttle_retry.py) that automatically retries failed API requests with exponential backoff while respecting rate limits across all music providers.**

The `music-assistant/server` repository isolates external service failures behind a unified error handling architecture. Every provider—including Spotify, Zvuk Music, Tidal, and YouTube—uses this framework to ensure robust API communication without overwhelming external services.

## Core Architecture of the Retry Framework

The framework consists of three coordinated components that handle rate limiting, error classification, and automatic retries.

### The Throttler Class

The **`Throttler`** class ([`music_assistant/helpers/throttle_retry.py`](https://github.com/music-assistant/server/blob/main/music_assistant/helpers/throttle_retry.py), lines 60-98) maintains a deque of timestamps (`_task_logs`) to enforce rate limits. When a provider calls `acquire()`, the class flushes entries older than the configured `period` and calculates the exact sleep time needed to respect the `rate_limit`.

```python
from music_assistant.helpers.throttle_retry import Throttler

# Limit to 5 calls per second

throttler = Throttler(rate_limit=5, period=1.0)

async def limited_call():
    delay = await throttler.acquire()  # Returns actual delay in seconds

    # Perform API request after throttling

```

The method returns the actual delay (0 seconds if no throttling occurred), enabling precise logging of provider wait times.

### ThrottlerManager and Retry Policies

The **`ThrottlerManager`** wraps the `Throttler` and configures retry behavior through two key parameters: `retry_attempts` and `initial_backoff`. It enforces maximum caps defined as `MAX_BACKOFF = 120` seconds and `MAX_RETRY_AFTER = 3600` seconds to prevent excessive waiting.

The manager provides two asynchronous context managers:

- **`acquire()`** – Standard throttling with retry logic
- **`bypass()`** – Sets the `BYPASS_THROTTLER` context variable to `True`, allowing internal operations to skip throttling

### The throttle_with_retries Decorator

The **`@throttle_with_retries`** decorator (implemented in `ThrottlerManager`) automates the entire retry lifecycle. When applied to provider methods, it:

1. Obtains a throttler slot via `acquire()`
2. Executes the wrapped coroutine
3. Catches `ResourceTemporarilyUnavailable` or `RateLimited` exceptions
4. Calculates backoff time (server-provided `Retry-After` or exponential backoff with jitter)
5. Retries up to `retry_attempts` times
6. Raises `RetriesExhausted` after final failure

```python
from music_assistant.helpers.throttle_retry import ThrottlerManager, throttle_with_retries

throttler = ThrottlerManager(
    rate_limit=10, 
    period=30,
    retry_attempts=5, 
    initial_backoff=5
)

class MyProvider:
    throttler: ThrottlerManager

    @throttle_with_retries
    async def fetch_something(self, resource_id: str) -> dict:
        # Automatically throttled and retried

        return await http_get_json(f"/resource/{resource_id}")

```

## Provider-Specific Error Mapping

Providers translate third-party library exceptions into Music Assistant's unified error hierarchy. This normalization allows `@throttle_with_retries` to handle retries generically across all services.

### Spotify HTTP Status Handling

In [`music_assistant/providers/spotify/provider.py`](https://github.com/music-assistant/server/blob/main/music_assistant/providers/spotify/provider.py) (lines 1191-1234), the `_get_data` and `_delete_data` methods map HTTP status codes to MA exception types:

- **429** → `RateLimited(backoff_time=int(Retry-After))`
- **502/503** → `ResourceTemporarilyUnavailable(backoff_time=30)`
- **401** → Token refresh triggered, then `ResourceTemporarilyUnavailable(backoff_time=1)`

### Zvuk Music Error Translation

The Zvuk Music provider implements a dedicated **`handle_zvuk_errors`** decorator in [`music_assistant/providers/zvuk_music/api_client.py`](https://github.com/music-assistant/server/blob/main/music_assistant/providers/zvuk_music/api_client.py) (lines 58-77). This wrapper catches `NetworkError`, `TimedOutError`, and `BadRequestError`, converting them to `ResourceTemporarilyUnavailable` or `RateLimited` as appropriate.

```python
@handle_zvuk_errors(not_found_return=None)
async def get_track(self, track_id: str) -> ZvukTrack | None:
    client = await self._get_client()
    return await client.get_track(track_id)

```

Other providers (Tidal, Apple Music, Yandex Music) follow this same pattern, ensuring consistent error semantics across the ecosystem.

## Implementing Automatic Retries in Providers

When a provider method raises `RateLimited` or `ResourceTemporarilyUnavailable`, the decorator initiates the retry cycle:

```python
@throttle_with_retries
async def _get_data(self, endpoint: str, **kwargs):
    # Request proceeds only after throttler slot acquisition

    # If temporary failure occurs:

    #   1. Log attempt with debug delay info

    #   2. Compute backoff (respect server Retry-After or use exponential)

    #   3. Sleep and retry (up to retry_attempts)

    #   4. Raise RetriesExhausted if all attempts fail

```

The framework automatically logs throttling delays at the `debug` level and backoff delays at the `info` level, providing visibility into provider performance.

## Bypassing Throttling for Internal Operations

Certain internal operations—such as cache warm-ups or in-memory state updates—must not be delayed by throttling. The **`BYPASS_THROTTLER`** context variable enables this exemption:

```python
async with self.throttler.bypass():
    # Code inside this block executes immediately

    await fast_internal_operation()

```

The `bypass()` context manager sets `BYPASS_THROTTLER` to `True`, causing `ThrottlerManager.acquire()` to yield a zero-delay slot for the duration of the block.

## Testing and Validation

The framework includes comprehensive test coverage in [`tests/core/test_throttle_retry.py`](https://github.com/music-assistant/server/blob/main/tests/core/test_throttle_retry.py), validating:

- Exponential backoff calculations
- Server-provided `Retry-After` handling
- Maximum backoff caps (`MAX_BACKOFF`)
- Retry exhaustion behavior (`RetriesExhausted`)

Provider-specific integration tests (e.g., [`tests/providers/spotify/test_api_client.py`](https://github.com/music-assistant/server/blob/main/tests/providers/spotify/test_api_client.py)) confirm that HTTP 429 responses correctly trigger the retry logic and eventually succeed after simulated backoff periods.

## Summary

- **Centralized framework**: All retry logic lives in [`music_assistant/helpers/throttle_retry.py`](https://github.com/music-assistant/server/blob/main/music_assistant/helpers/throttle_retry.py), ensuring consistency across providers.
- **Automatic retries**: The `@throttle_with_retries` decorator handles `RateLimited` and `ResourceTemporarilyUnavailable` exceptions with exponential backoff and jitter.
- **Provider agnostic**: Providers map third-party errors to MA exception types, allowing the retry framework to work uniformly across Spotify, Zvuk Music, Tidal, and others.
- **Configurable limits**: `ThrottlerManager` enforces `MAX_BACKOFF` (120s) and `MAX_RETRY_AFTER` (3600s) to prevent indefinite blocking.
- **Internal bypass**: The `BYPASS_THROTTLER` context variable allows critical internal operations to skip throttling when necessary.

## Frequently Asked Questions

### How does Music Assistant handle rate limiting from music providers?

Music Assistant uses the `Throttler` class to track API call timestamps and enforce rate limits before requests are sent. When a provider returns HTTP 429 (Rate Limited), the `ThrottlerManager` extracts the `Retry-After` header and uses it as the minimum backoff time, adding jitter to prevent thundering herd problems.

### What happens when all retry attempts are exhausted?

After exceeding the configured `retry_attempts` (typically 5), the framework raises `RetriesExhausted`. This exception propagates to the caller, indicating that the provider is unavailable and manual intervention or later retry may be required. The failure is logged with the total duration and number of attempts made.

### Can providers disable throttling for specific operations?

Yes. Providers can use `async with self.throttler.bypass():` to execute code without throttling delays. This is useful for internal housekeeping tasks like cache updates or token refresh operations that should not count against external API quotas. The bypass uses a context variable to ensure thread-safe operation across async tasks.

### How do provider-specific errors integrate with the retry system?

Providers implement error mapping wrappers (like `handle_zvuk_errors` for Zvuk Music) that catch library-specific exceptions (`NetworkError`, `TimedOutError`) and re-raise them as `ResourceTemporarilyUnavailable` or `RateLimited`. This translation layer allows the centralized `@throttle_with_retries` decorator to handle retries without knowing the specifics of each third-party API.