# How Pixelle-Video Performs Voice Cloning with Reference Audio: A Complete Technical Guide

> Discover how Pixelle-Video performs voice cloning using reference audio. This technical guide details the ComfyUI TTS pipeline and speaker adaptation for realistic voice generation. Learn more.

- Repository: [AIDC-AI/Pixelle-Video](https://github.com/AIDC-AI/Pixelle-Video)
- Tags: deep-dive
- Published: 2026-04-23

---

**Pixelle-Video implements voice cloning by passing a user-uploaded reference audio file through a ComfyUI-based TTS pipeline, where an external workflow executes the actual speaker adaptation algorithm.**

Voice cloning in generative video workflows enables personalized digital human creation without requiring professional voice recording equipment. This article examines how the open-source **Pixelle-Video** repository (AIDC-AI/Pixelle-Video) implements this capability through reference audio processing, tracing the complete data flow from file upload to synthesized speech output.

## Reference Audio Upload and Temporary Storage

The voice cloning process begins in the Streamlit-based UI component defined in [`web/components/digital_tts_config.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/web/components/digital_tts_config.py). This module provides a file uploader specifically configured for reference audio files:

```python

# web/components/digital_tts_config.py

ref_audio_file = st.file_uploader(
    tr("tts.ref_audio"),
    type=["mp3", "wav", "flac", "m4a", "aac", "ogg"],
    key="digital_ref_audio_upload",
)

if ref_audio_file is not None:
    temp_dir = Path("temp")
    temp_dir.mkdir(exist_ok=True)
    ref_audio_path = temp_dir / f"ref_audio_{ref_audio_file.name}"
    with open(ref_audio_path, "wb") as f:
        f.write(ref_audio_file.getbuffer())

```

The component accepts common audio formats including MP3, WAV, FLAC, and AAC. Upon upload, the file is written to a temporary location at `temp/ref_audio_<filename>` and the absolute path is stored in `ref_audio_path` for downstream processing.

## Propagating Reference Audio Through the Digital Human Pipeline

The [`web/pipelines/digital_human.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/web/pipelines/digital_human.py) module orchestrates the complete video generation workflow. After gathering parameters from the UI, it constructs a `video_params` dictionary that includes the reference audio path when present:

```python

# web/pipelines/digital_human.py (parameter extraction)

video_params = {
    "text": generated_text,
    "audio_path": audio_path,
    "tts_inference_mode": tts_inference_mode,
    # ... other parameters

}

if ref_audio_path:
    video_params["ref_audio"] = ref_audio_path

```

When the pipeline reaches the TTS generation step, it builds specialized arguments for the TTS service call. The reference audio path is conditionally included based on the inference mode:

```python

# web/pipelines/digital_human.py (TTS invocation)

tts_kwargs = {
    "text": generated_text,
    "output_path": audio_path,
    "inference_mode": tts_inference_mode,
}

if tts_inference_mode == "comfyui":
    if tts_workflow:
        tts_kwargs["workflow"] = tts_workflow
    if ref_audio:                # <-- reference audio path forwarded

        tts_kwargs["ref_audio"] = ref_audio

await pixelle_video.tts(**tts_kwargs)

```

## TTS Service Routing and ComfyUI Integration

The [`pixelle_video/services/tts_service.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/pixelle_video/services/tts_service.py) module handles the actual TTS execution. It supports multiple inference backends, with ComfyUI being the primary mode for voice cloning functionality.

The `TTSService.__call__` method routes requests based on the `inference_mode` parameter:

```python

# pixelle_video/services/tts_service.py (simplified routing)

async def __call__(self, text: str, output_path: str, inference_mode: str = "local", **params):
    if inference_mode == "comfyui":
        return await self._call_comfyui_workflow(text, output_path, **params)
    # ... local TTS handling

```

The ComfyUI workflow execution method receives all parameters from the pipeline, including the reference audio path:

```python

# pixelle_video/services/tts_service.py

async def _call_comfyui_workflow(self, text: str, output_path: str, workflow: str = None, **params):
    # Build base workflow parameters

    workflow_params = {"text": text}
    
    # Merge any additional parameters, including ref_audio

    workflow_params.update(params)
    
    # Execute workflow through ComfyUI/RunningHub API

    result = await kit.execute(workflow_input, workflow_params)
    return result

```

## How the ComfyUI Workflow Executes Voice Cloning

Pixelle-Video does not implement the voice cloning algorithm directly. Instead, it delegates to **ComfyUI workflows** that execute the actual speaker adaptation. The workflow JSON (typically sourced from RunningHub or custom definitions) expects specific input parameters:

| Parameter | Description |
|-----------|-------------|
| `text` | Target text to synthesize |
| `ref_audio` | Path to reference audio file for voice cloning |
| `workflow` | Identifier for the specific ComfyUI workflow configuration |

The workflow internally uses voice cloning models such as **RVC (Retrieval-based Voice Conversion)** or **SoftVC-based VITS** to:

1. Extract speaker characteristics from the reference audio
2. Generate base speech from the input text using a foundation TTS model
3. Apply speaker conversion to match the reference voice characteristics
4. Return the synthesized audio file

The result is passed back through the TTS service to the pipeline, where it is synchronized with the generated video.

## Configuration Requirements for Voice Cloning

The [`pixelle_video/config/manager.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/pixelle_video/config/manager.py) module manages ComfyUI connectivity settings required for voice cloning functionality:

| Setting | Purpose |
|---------|---------|
| `RUNNINGHUB_API_KEY` | Authentication for RunningHub-hosted ComfyUI workflows |
| `RUNNINGHUB_WORKFLOW_URL` | Endpoint for workflow execution |
| `COMFYUI_WORKFLOWS` | Registry of available workflow configurations |

These settings enable the TTS service to communicate with external ComfyUI instances where the actual voice cloning computation occurs.

## Summary

Pixelle-Video implements voice cloning through a clean separation between pipeline orchestration and model execution:

- **UI upload**: [`web/components/digital_tts_config.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/web/components/digital_tts_config.py) handles reference audio file upload and temporary storage
- **Pipeline propagation**: [`web/pipelines/digital_human.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/web/pipelines/digital_human.py) passes the audio path through `video_params` to the TTS service
- **Service routing**: [`pixelle_video/services/tts_service.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/pixelle_video/services/tts_service.py) directs ComfyUI-mode requests to external workflow execution
- **Model execution**: ComfyUI workflows perform the actual voice cloning using speaker adaptation algorithms

This architecture allows Pixelle-Video to leverage state-of-the-art voice cloning models without maintaining them directly in the repository.

## Frequently Asked Questions

### What audio formats are supported for voice cloning reference audio?

Pixelle-Video accepts MP3, WAV, FLAC, M4A, AAC, and OGG files through the reference audio uploader in [`web/components/digital_tts_config.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/web/components/digital_tts_config.py). The file is automatically saved to a temporary directory before processing.

### Does Pixelle-Video implement its own voice cloning algorithm?

No. Pixelle-Video delegates voice cloning to **ComfyUI workflows** that run externally. The repository handles file management, parameter passing, and workflow orchestration, while the actual speaker adaptation uses models like RVC or VITS variants implemented in the ComfyUI workflow JSON.

### Why is my reference audio not affecting the output voice?

Ensure your TTS **inference mode** is set to `"comfyui"` in the pipeline configuration. The reference audio parameter is only forwarded when this mode is active, as shown in [`web/pipelines/digital_human.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/web/pipelines/digital_human.py). Additionally, verify that your ComfyUI workflow JSON actually consumes the `ref_audio` parameter—some workflows may ignore it or use different parameter names.

### Can I use local voice cloning models instead of ComfyUI?

The current implementation in [`pixelle_video/services/tts_service.py`](https://github.com/AIDC-AI/Pixelle-Video/blob/main/pixelle_video/services/tts_service.py) primarily supports `"local"` and `"comfyui"` inference modes. The `"local"` mode does not implement voice cloning in the analyzed codebase—it generates standard TTS without speaker adaptation. For voice cloning functionality, ComfyUI mode with an appropriate workflow is required.