How Pixelle-Video Performs Voice Cloning with Reference Audio: A Complete Technical Guide

Pixelle-Video implements voice cloning by passing a user-uploaded reference audio file through a ComfyUI-based TTS pipeline, where an external workflow executes the actual speaker adaptation algorithm.

Voice cloning in generative video workflows enables personalized digital human creation without requiring professional voice recording equipment. This article examines how the open-source Pixelle-Video repository (AIDC-AI/Pixelle-Video) implements this capability through reference audio processing, tracing the complete data flow from file upload to synthesized speech output.

Reference Audio Upload and Temporary Storage

The voice cloning process begins in the Streamlit-based UI component defined in web/components/digital_tts_config.py. This module provides a file uploader specifically configured for reference audio files:

# web/components/digital_tts_config.py

ref_audio_file = st.file_uploader(
    tr("tts.ref_audio"),
    type=["mp3", "wav", "flac", "m4a", "aac", "ogg"],
    key="digital_ref_audio_upload",
)

if ref_audio_file is not None:
    temp_dir = Path("temp")
    temp_dir.mkdir(exist_ok=True)
    ref_audio_path = temp_dir / f"ref_audio_{ref_audio_file.name}"
    with open(ref_audio_path, "wb") as f:
        f.write(ref_audio_file.getbuffer())

The component accepts common audio formats including MP3, WAV, FLAC, and AAC. Upon upload, the file is written to a temporary location at temp/ref_audio_<filename> and the absolute path is stored in ref_audio_path for downstream processing.

Propagating Reference Audio Through the Digital Human Pipeline

The web/pipelines/digital_human.py module orchestrates the complete video generation workflow. After gathering parameters from the UI, it constructs a video_params dictionary that includes the reference audio path when present:

# web/pipelines/digital_human.py (parameter extraction)

video_params = {
    "text": generated_text,
    "audio_path": audio_path,
    "tts_inference_mode": tts_inference_mode,
    # ... other parameters

}

if ref_audio_path:
    video_params["ref_audio"] = ref_audio_path

When the pipeline reaches the TTS generation step, it builds specialized arguments for the TTS service call. The reference audio path is conditionally included based on the inference mode:

# web/pipelines/digital_human.py (TTS invocation)

tts_kwargs = {
    "text": generated_text,
    "output_path": audio_path,
    "inference_mode": tts_inference_mode,
}

if tts_inference_mode == "comfyui":
    if tts_workflow:
        tts_kwargs["workflow"] = tts_workflow
    if ref_audio:                # <-- reference audio path forwarded

        tts_kwargs["ref_audio"] = ref_audio

await pixelle_video.tts(**tts_kwargs)

TTS Service Routing and ComfyUI Integration

The pixelle_video/services/tts_service.py module handles the actual TTS execution. It supports multiple inference backends, with ComfyUI being the primary mode for voice cloning functionality.

The TTSService.__call__ method routes requests based on the inference_mode parameter:

# pixelle_video/services/tts_service.py (simplified routing)

async def __call__(self, text: str, output_path: str, inference_mode: str = "local", **params):
    if inference_mode == "comfyui":
        return await self._call_comfyui_workflow(text, output_path, **params)
    # ... local TTS handling

The ComfyUI workflow execution method receives all parameters from the pipeline, including the reference audio path:

# pixelle_video/services/tts_service.py

async def _call_comfyui_workflow(self, text: str, output_path: str, workflow: str = None, **params):
    # Build base workflow parameters

    workflow_params = {"text": text}
    
    # Merge any additional parameters, including ref_audio

    workflow_params.update(params)
    
    # Execute workflow through ComfyUI/RunningHub API

    result = await kit.execute(workflow_input, workflow_params)
    return result

How the ComfyUI Workflow Executes Voice Cloning

Pixelle-Video does not implement the voice cloning algorithm directly. Instead, it delegates to ComfyUI workflows that execute the actual speaker adaptation. The workflow JSON (typically sourced from RunningHub or custom definitions) expects specific input parameters:

Parameter	Description
text	Target text to synthesize
ref_audio	Path to reference audio file for voice cloning
workflow	Identifier for the specific ComfyUI workflow configuration

The workflow internally uses voice cloning models such as RVC (Retrieval-based Voice Conversion) or SoftVC-based VITS to:

1. Extract speaker characteristics from the reference audio
2. Generate base speech from the input text using a foundation TTS model
3. Apply speaker conversion to match the reference voice characteristics
4. Return the synthesized audio file

The result is passed back through the TTS service to the pipeline, where it is synchronized with the generated video.

Configuration Requirements for Voice Cloning

The pixelle_video/config/manager.py module manages ComfyUI connectivity settings required for voice cloning functionality:

Setting	Purpose
RUNNINGHUB_API_KEY	Authentication for RunningHub-hosted ComfyUI workflows
RUNNINGHUB_WORKFLOW_URL	Endpoint for workflow execution
COMFYUI_WORKFLOWS	Registry of available workflow configurations

These settings enable the TTS service to communicate with external ComfyUI instances where the actual voice cloning computation occurs.

Summary

Pixelle-Video implements voice cloning through a clean separation between pipeline orchestration and model execution:

• UI upload: web/components/digital_tts_config.py handles reference audio file upload and temporary storage
• Pipeline propagation: web/pipelines/digital_human.py passes the audio path through video_params to the TTS service
• Service routing: pixelle_video/services/tts_service.py directs ComfyUI-mode requests to external workflow execution
• Model execution: ComfyUI workflows perform the actual voice cloning using speaker adaptation algorithms

This architecture allows Pixelle-Video to leverage state-of-the-art voice cloning models without maintaining them directly in the repository.

Frequently Asked Questions

What audio formats are supported for voice cloning reference audio?

Pixelle-Video accepts MP3, WAV, FLAC, M4A, AAC, and OGG files through the reference audio uploader in web/components/digital_tts_config.py. The file is automatically saved to a temporary directory before processing.

Does Pixelle-Video implement its own voice cloning algorithm?

No. Pixelle-Video delegates voice cloning to ComfyUI workflows that run externally. The repository handles file management, parameter passing, and workflow orchestration, while the actual speaker adaptation uses models like RVC or VITS variants implemented in the ComfyUI workflow JSON.

Why is my reference audio not affecting the output voice?

Ensure your TTS inference mode is set to "comfyui" in the pipeline configuration. The reference audio parameter is only forwarded when this mode is active, as shown in web/pipelines/digital_human.py. Additionally, verify that your ComfyUI workflow JSON actually consumes the ref_audio parameter—some workflows may ignore it or use different parameter names.

Can I use local voice cloning models instead of ComfyUI?

The current implementation in pixelle_video/services/tts_service.py primarily supports "local" and "comfyui" inference modes. The "local" mode does not implement voice cloning in the analyzed codebase—it generates standard TTS without speaker adaptation. For voice cloning functionality, ComfyUI mode with an appropriate workflow is required.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}