How to Use Demucs for Audio Source Separation in the ailia‑models Repository

The Demucs implementation in ailia‑models separates stereo audio into four stems—drums, bass, other, and vocals—using chunk‑based inference with optional ONNX Runtime support.

The ailia‑models repository by axinc‑ai provides a ready‑to‑use pipeline for music source separation based on Facebook Research’s Demucs architecture. This guide explains how to run inference using the htdemucs_ft model, configure runtime backends, and process audio programmatically or via CLI.

Model Architecture and Weights

The Demucs implementation defines a htdemucs_ft model composed of four distinct ONNX networks—one for each target source (drums, bass, other, vocals)—and a segment length expressed as a Fraction in the initialization code.

In audio_processing/demucs/demucs.py, the main() function handles model setup between lines 45–63. It first invokes check_and_download_models (imported from util/model_utils.py) to fetch the .onnx and .prototxt files from a Google Storage bucket on first run (lines 65–70). Internet access is required only for this initial download.

Runtime Selection: Ailia SDK vs. ONNX Runtime

You can choose between two inference backends via the --onnx CLI flag:

• Ailia SDK (default): When --onnx is omitted, the script loads each source network using ailia.Net with an env_id derived from CLI arguments (lines 73–94).
• ONNX Runtime: When --onnx is supplied, the script creates an onnxruntime.InferenceSession for each of the four source models, removing the dependency on the ailia SDK.

Use the ONNX Runtime option for environments where the ailia SDK is not installed or when debugging model behavior in pure ONNX tools.

Audio Loading and Preprocessing

The load_audio function (lines 81–103) reads input files using librosa, with an optional ffmpeg fallback if enabled. It automatically resamples audio to 44.1 kHz and ensures stereo output by duplicating mono channels, guaranteeing a tensor shape of (2, T) where T is the number of samples. This normalization step is critical because the Demucs model expects standard CD‑quality stereo input.

Chunk‑Based Inference Pipeline

The core separation logic resides in apply_model (lines 124–152), which processes long audio files in overlapping chunks to manage memory usage and maintain temporal coherence.

Key parameters controlling this behavior include:

• segment: Length of each analysis window.
• overlap: Fractional overlap between consecutive chunks.
• transition_power: Exponent used for the weighted triangular window that blends chunk outputs.
• shifts: Optional random temporal offsets applied to the input; results are averaged across shifts to reduce boundary artifacts.

When split=True, the algorithm iterates over the mixture using the weighted window to smoothly merge predictions from adjacent segments.

Source Isolation and Output Generation

Inside the predict function (lines 75–99), the pipeline isolates each source by temporarily swapping the target network into models["net"], running apply_model, and zeroing out the outputs for all other sources before accumulation. After processing all four networks, the final estimates are denormalized to the original audio scale to preserve input loudness levels.

Results are saved using either:

• MP3: If the optional lameenc encoder is available.
• WAV: Fallback to soundfile for uncompressed PCM output.

Command‑Line Usage

The script exposes standard arguments through get_base_parser and update_parser (from util/arg_utils.py), including --input, --savepath, --model_type, and --onnx.

Separate the default test audio into four stems:

python3 audio_processing/demucs/demucs.py

Process a custom file with ONNX Runtime and specify an output directory:

python3 audio_processing/demucs/demucs.py \
  --input path/to/song.wav \
  --savepath results/ \
  --onnx

Programmatic Integration

You can invoke the pipeline from Python by importing the main entry point:

from audio_processing.demucs.demucs import main as demucs_main

# Executes with sys.argv; handles all CLI arguments internally

demucs_main()

For advanced use cases involving NumPy arrays directly, you would replicate the initialization steps from main() to populate the models dictionary, then call predict(models, audio_array). Note that load_audio returns a NumPy array of shape (2, T) ready for inference.

Summary

• The Demucs model in audio_processing/demucs/demucs.py separates stereo audio into drums, bass, other, and vocals using four independent ONNX networks.
• Automatic weight downloads via check_and_download_models require internet only on first run.
• Use the --onnx flag to switch from the Ailia SDK to ONNX Runtime for inference.
• Audio is resampled to 44.1 kHz and processed in overlapping chunks with optional shifts to improve quality.
• Output files are written as MP3 (with lameenc) or WAV (with soundfile) based on encoder availability.

Frequently Asked Questions

What audio file formats are supported as input?

The load_audio function relies on librosa, which supports WAV, MP3, FLAC, and OGG. If ffmpeg is installed and enabled, the function can also decode formats outside librosa’s native support, provided they can be resampled to 44.1 kHz stereo.

Can I run the model without installing the ailia SDK?

Yes. Supply the --onnx flag when executing demucs.py. This forces the script to use onnxruntime.InferenceSession instead of ailia.Net, allowing the pipeline to run on systems where only ONNX Runtime is available.

How does the chunk‑based processing handle long audio files?

The apply_model function divides the input into segments defined by the segment parameter, processes each chunk separately, and blends overlapping regions using a weighted triangular window controlled by transition_power. Optional shifts average multiple temporal offsets to minimize audible artifacts at chunk boundaries.

Why are there four separate ONNX models instead of one?

The htdemucs_ft architecture uses source‑specific networks to isolate drums, bass, other, and vocals independently. During predict, each network is loaded sequentially as models["net"], and the outputs for non‑target sources are masked to zero before accumulation, enabling precise stem extraction without cross‑source interference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}