How the Audio Isolation Feature Removes Background Noise from Recordings

The audio isolation feature removes background noise by streaming audio files to the ElevenLabs Audio Isolation API through a Kotlin Multiplatform abstraction layer, then persisting the cleaned audio bytes returned by the cloud service.

The audio isolation feature in the Muse repository provides a server-driven approach to denoising recordings without requiring on-device machine learning. Implemented as a Kotlin Multiplatform module, the feature delegates complex signal processing to ElevenLabs' cloud API while maintaining a consistent contract across Android and iOS platforms. This architecture ensures that background noise removal behaves identically regardless of the underlying operating system.

Architecture of the Audio Isolation System

The implementation follows a layered design that separates UI concerns from core processing and external API integration.

The Provider Contract

At the foundation lies the AudioIsolationProvider interface. This contract declares the single responsibility of noise removal:

interface AudioIsolationProvider {
    suspend fun removeBackgroundNoise(
        audio: Source,
        audioName: String
    ): Result<ByteArray>
}

Any concrete implementation must accept an Okio Source containing the raw audio and return a Result wrapping the cleaned bytes. The repository provides two implementations: the production ElevenLabsClient and a MockAudioIsolationProvider for testing.

The Processing Manager

The SpeechProcessorManager serves as the orchestration layer. It handles platform-specific file operations, invokes the provider, and manages persistence. Both iOS and Android variants implement removeBackgroundNoiseAndSave(), which ultimately calls the provider's removeBackgroundNoise() method.

In [SpeechProcessorManager.ios.kt](https://github.com/kkoshin/muse/blob/main/muse/src/iosMain/kotlin/io/github/kkoshin/muse/core/manager/SpeechProcessorManager.ios.kt), the implementation opens the audio file as a Source and forwards it:

actual suspend fun removeBackgroundNoise(audioUri: Path): Result<ByteArray> {
    return withContext(Dispatchers.IO) {
        val name = audioUri.name
        isolationProvider.removeBackgroundNoise(
            FileSystem.SYSTEM.source(audioUri).buffer(),
            name
        )
    }
}

The Android counterpart follows an identical pattern, ensuring behavioral consistency across platforms.

The ElevenLabs API Client

The concrete implementation resides in [AudioIsolation.kt](https://github.com/kkoshin/muse/blob/main/elevenlabs/src/commonMain/kotlin/io/github/kkoshin/elevenlabs/api/AudioIsolation.kt). The removeBackgroundAudio() function constructs a multipart POST request to the /audio-isolation endpoint:

suspend fun ElevenLabsClient.removeBackgroundAudio(
    audio: Source,
    audioName: String,
): Result<ByteArray> = postForm(AudioIsolation()) {
    append(
        "audio",
        audio.buffer().readByteArray(),
        headers = Headers.build {
            append(HttpHeaders.ContentDisposition,
                "form-data; name=\"audio\"; filename=\"$audioName\"")
            append(HttpHeaders.ContentType, ContentType.Audio.Any.toString())
        }
    )
}

The server processes the upload, isolates the primary audio signal, and returns the denoised content as a raw ByteArray.

Execution Flow from UI to API

The complete workflow traverses four distinct layers:

1. User Interaction: [AudioIsolationScreen.kt](https://github.com/kkoshin/muse/blob/main/muse/src/commonMain/kotlin/io/github/kkoshin/muse/feature/isolation/AudioIsolationScreen.kt) captures the user action and invokes AudioIsolationViewModel.removeBackgroundNoiseAndSave().

2. ViewModel Coordination: The ViewModel calls SpeechProcessorManager.removeBackgroundNoiseAndSave(), passing the source audio URI.

3. Provider Execution: The manager converts the URI to an Okio Source and calls isolationProvider.removeBackgroundNoise(), which transmits the data to ElevenLabs.

4. Persistence: Upon receiving the cleaned bytes, the manager writes the output to the device's Downloads folder, returning the final file path to the UI.

Platform Implementations and Testing

iOS and Android Specifics

Both platforms share the same SpeechProcessorManager interface but provide platform-specific actual implementations for file system access. The iOS version uses FileSystem.SYSTEM.source() to read the audio file, while the Android version handles content provider resolution. Despite these differences, both call the identical AudioIsolationProvider contract, keeping the noise removal logic platform-agnostic.

Mock Provider for Development

For debugging and offline testing, the MockAudioIsolationProvider class implements the interface by returning a static MP3 asset. This allows developers to test the full UI flow without consuming API credits or requiring network connectivity.

Summary

• The audio isolation feature relies on the AudioIsolationProvider interface to abstract noise removal operations across platforms.
• SpeechProcessorManager implementations for iOS and Android handle file I/O and delegate processing to the provider.
• The actual denoising occurs via the ElevenLabs API in [AudioIsolation.kt](https://github.com/kkoshin/muse/blob/main/elevenlabs/src/commonMain/kotlin/io/github/kkoshin/elevenlabs/api/AudioIsolation.kt), which streams audio to the /audio-isolation endpoint.
• The cleaned audio returns as a ByteArray and saves to the Downloads folder for user access.
• A MockAudioIsolationProvider enables offline testing by returning static audio assets.

Frequently Asked Questions

What external service powers the audio isolation feature?

The feature uses the ElevenLabs Audio Isolation API, accessed through the ElevenLabsClient.removeBackgroundAudio() function in the ElevenLabs SDK module. This cloud service handles the complex machine learning models required to separate voice from background noise.

How does Muse ensure consistent behavior across iOS and Android?

Muse utilizes Kotlin Multiplatform with a shared AudioIsolationProvider interface. The SpeechProcessorManager provides platform-specific actual implementations for file handling, but both platforms invoke the same provider method, ensuring identical noise removal results regardless of device.

Can developers test the audio isolation feature without an internet connection?

Yes. The repository includes a MockAudioIsolationProvider implementation that returns a static MP3 file from local assets. This mock provider implements the same interface as the production ElevenLabs client, allowing UI testing and development without network requests or API keys.

Where does the application store the cleaned audio files?

After the ElevenLabs API returns the processed bytes, the SpeechProcessorManager writes the data to the device's Downloads folder using mediaStoreHelper.exportFileToDownload(). This makes the denoised recording immediately accessible through the system's Downloads app or file manager.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}