Provider Error Handling and Automatic Retry Mechanisms in Music Assistant

Music Assistant implements a centralized throttling and retry framework in music_assistant/helpers/throttle_retry.py that automatically retries failed API requests with exponential backoff while respecting rate limits across all music providers.

The music-assistant/server repository isolates external service failures behind a unified error handling architecture. Every provider—including Spotify, Zvuk Music, Tidal, and YouTube—uses this framework to ensure robust API communication without overwhelming external services.

Core Architecture of the Retry Framework

The framework consists of three coordinated components that handle rate limiting, error classification, and automatic retries.

The Throttler Class

The Throttler class (music_assistant/helpers/throttle_retry.py, lines 60-98) maintains a deque of timestamps (_task_logs) to enforce rate limits. When a provider calls acquire(), the class flushes entries older than the configured period and calculates the exact sleep time needed to respect the rate_limit.

from music_assistant.helpers.throttle_retry import Throttler

# Limit to 5 calls per second

throttler = Throttler(rate_limit=5, period=1.0)

async def limited_call():
    delay = await throttler.acquire()  # Returns actual delay in seconds

    # Perform API request after throttling

The method returns the actual delay (0 seconds if no throttling occurred), enabling precise logging of provider wait times.

ThrottlerManager and Retry Policies

The ThrottlerManager wraps the Throttler and configures retry behavior through two key parameters: retry_attempts and initial_backoff. It enforces maximum caps defined as MAX_BACKOFF = 120 seconds and MAX_RETRY_AFTER = 3600 seconds to prevent excessive waiting.

The manager provides two asynchronous context managers:

• acquire() – Standard throttling with retry logic
• bypass() – Sets the BYPASS_THROTTLER context variable to True, allowing internal operations to skip throttling

The throttle_with_retries Decorator

The @throttle_with_retries decorator (implemented in ThrottlerManager) automates the entire retry lifecycle. When applied to provider methods, it:

1. Obtains a throttler slot via acquire()
2. Executes the wrapped coroutine
3. Catches ResourceTemporarilyUnavailable or RateLimited exceptions
4. Calculates backoff time (server-provided Retry-After or exponential backoff with jitter)
5. Retries up to retry_attempts times
6. Raises RetriesExhausted after final failure

from music_assistant.helpers.throttle_retry import ThrottlerManager, throttle_with_retries

throttler = ThrottlerManager(
    rate_limit=10, 
    period=30,
    retry_attempts=5, 
    initial_backoff=5
)

class MyProvider:
    throttler: ThrottlerManager

    @throttle_with_retries
    async def fetch_something(self, resource_id: str) -> dict:
        # Automatically throttled and retried

        return await http_get_json(f"/resource/{resource_id}")

Provider-Specific Error Mapping

Providers translate third-party library exceptions into Music Assistant's unified error hierarchy. This normalization allows @throttle_with_retries to handle retries generically across all services.

Spotify HTTP Status Handling

In music_assistant/providers/spotify/provider.py (lines 1191-1234), the _get_data and _delete_data methods map HTTP status codes to MA exception types:

• 429 → RateLimited(backoff_time=int(Retry-After))
• 502/503 → ResourceTemporarilyUnavailable(backoff_time=30)
• 401 → Token refresh triggered, then ResourceTemporarilyUnavailable(backoff_time=1)

Zvuk Music Error Translation

The Zvuk Music provider implements a dedicated handle_zvuk_errors decorator in music_assistant/providers/zvuk_music/api_client.py (lines 58-77). This wrapper catches NetworkError, TimedOutError, and BadRequestError, converting them to ResourceTemporarilyUnavailable or RateLimited as appropriate.

@handle_zvuk_errors(not_found_return=None)
async def get_track(self, track_id: str) -> ZvukTrack | None:
    client = await self._get_client()
    return await client.get_track(track_id)

Other providers (Tidal, Apple Music, Yandex Music) follow this same pattern, ensuring consistent error semantics across the ecosystem.

Implementing Automatic Retries in Providers

When a provider method raises RateLimited or ResourceTemporarilyUnavailable, the decorator initiates the retry cycle:

@throttle_with_retries
async def _get_data(self, endpoint: str, **kwargs):
    # Request proceeds only after throttler slot acquisition

    # If temporary failure occurs:

    #   1. Log attempt with debug delay info

    #   2. Compute backoff (respect server Retry-After or use exponential)

    #   3. Sleep and retry (up to retry_attempts)

    #   4. Raise RetriesExhausted if all attempts fail

The framework automatically logs throttling delays at the debug level and backoff delays at the info level, providing visibility into provider performance.

Bypassing Throttling for Internal Operations

Certain internal operations—such as cache warm-ups or in-memory state updates—must not be delayed by throttling. The BYPASS_THROTTLER context variable enables this exemption:

async with self.throttler.bypass():
    # Code inside this block executes immediately

    await fast_internal_operation()

The bypass() context manager sets BYPASS_THROTTLER to True, causing ThrottlerManager.acquire() to yield a zero-delay slot for the duration of the block.

Testing and Validation

The framework includes comprehensive test coverage in tests/core/test_throttle_retry.py, validating:

• Exponential backoff calculations
• Server-provided Retry-After handling
• Maximum backoff caps (MAX_BACKOFF)
• Retry exhaustion behavior (RetriesExhausted)

Provider-specific integration tests (e.g., tests/providers/spotify/test_api_client.py) confirm that HTTP 429 responses correctly trigger the retry logic and eventually succeed after simulated backoff periods.

Summary

• Centralized framework: All retry logic lives in music_assistant/helpers/throttle_retry.py, ensuring consistency across providers.
• Automatic retries: The @throttle_with_retries decorator handles RateLimited and ResourceTemporarilyUnavailable exceptions with exponential backoff and jitter.
• Provider agnostic: Providers map third-party errors to MA exception types, allowing the retry framework to work uniformly across Spotify, Zvuk Music, Tidal, and others.
• Configurable limits: ThrottlerManager enforces MAX_BACKOFF (120s) and MAX_RETRY_AFTER (3600s) to prevent indefinite blocking.
• Internal bypass: The BYPASS_THROTTLER context variable allows critical internal operations to skip throttling when necessary.

Frequently Asked Questions

How does Music Assistant handle rate limiting from music providers?

Music Assistant uses the Throttler class to track API call timestamps and enforce rate limits before requests are sent. When a provider returns HTTP 429 (Rate Limited), the ThrottlerManager extracts the Retry-After header and uses it as the minimum backoff time, adding jitter to prevent thundering herd problems.

What happens when all retry attempts are exhausted?

After exceeding the configured retry_attempts (typically 5), the framework raises RetriesExhausted. This exception propagates to the caller, indicating that the provider is unavailable and manual intervention or later retry may be required. The failure is logged with the total duration and number of attempts made.

Can providers disable throttling for specific operations?

Yes. Providers can use async with self.throttler.bypass(): to execute code without throttling delays. This is useful for internal housekeeping tasks like cache updates or token refresh operations that should not count against external API quotas. The bypass uses a context variable to ensure thread-safe operation across async tasks.

How do provider-specific errors integrate with the retry system?

Providers implement error mapping wrappers (like handle_zvuk_errors for Zvuk Music) that catch library-specific exceptions (NetworkError, TimedOutError) and re-raise them as ResourceTemporarilyUnavailable or RateLimited. This translation layer allows the centralized @throttle_with_retries decorator to handle retries without knowing the specifics of each third-party API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}