DeepFilterNet Audio Enhancement Conflicts with Pocket TTS in Speech-to-Speech Pipeline

DeepFilterNet audio enhancement and Pocket TTS are mutually exclusive in the huggingface/speech-to-speech pipeline because they apply incompatible resampling and processing expectations to the same audio stream.

The huggingface/speech-to-speech repository provides a modular pipeline for real-time voice conversations, optionally integrating DeepFilterNet (DFN) for microphone noise suppression and Pocket TTS for streaming text-to-speech synthesis. However, enabling both features simultaneously creates a technical conflict where the VAD handler attempts to enhance synthetic TTS audio as if it were raw microphone input. Understanding this DeepFilterNet audio enhancement conflict with Pocket TTS is essential for configuring stable production deployments.

Why DeepFilterNet and Pocket TTS Conflict

The pipeline treats both microphone input and TTS-generated audio as identical VADAudio objects, causing the Voice Activity Detection (VAD) handler to apply DeepFilterNet enhancement indiscriminately.

How DeepFilterNet Processes Audio

In src/speech_to_speech/VAD/vad_handler.py, the optional audio enhancement is wired through the audio_enhancement boolean flag (lines 70–74). When enabled, the _apply_audio_enhancement method (lines 700–702) receives the final VAD-emitted audio chunk as a continuous NumPy waveform.

The DFN model expects a single, continuous speech recording and internally uses torchaudio.functional.resample to align the sample rate with self.df_state.sr(). This processing assumes the input is genuine microphone capture requiring noise suppression, not synthetic speech.

How Pocket TTS Handles Audio

The PocketTTSHandler in src/speech_to_speech/TTS/pocket_tts_handler.py (lines 70–87 and 175–200) generates audio in small blocks (approximately 10 ms) that are pre-resampled to the pipeline's target sample_rate (default 16 kHz) before yielding. The setup method accepts parameters including sample_rate: int = 16000 and blocksize: int = 512, creating a streaming output that is already optimized for playback.

The Root Cause of the Incompatibility

The conflict arises because audio_enhancement is a global VAD configuration that does not discriminate between audio sources. When Pocket TTS is active, the VAD handler still invokes _apply_audio_enhancement on the final chunk containing TTS-generated audio.

This creates two problems:

1. Domain mismatch: DeepFilterNet is trained for natural speech recorded on microphones, not synthetic TTS output, leading to potential shape mismatches or unnecessary computation.
2. Double resampling: Since Pocket TTS already resamples audio to 16 kHz (or the configured sample_rate), the additional torchaudio resample step inside DFN misaligns the waveform length and corrupts the audio stream.

How to Resolve the Conflict

You must ensure that DeepFilterNet enhancement runs only on microphone input, not on TTS-generated streams.

Disable Audio Enhancement When Using Pocket TTS

The simplest solution is to set audio_enhancement=False in your VADHandlerArguments when instantiating the pipeline with Pocket TTS enabled.

Use Separate Pipelines

If your application requires both features in the same session, run two distinct pipeline instances: one for the STT (speech-to-text) path with DFN enabled for microphone cleaning, and a separate one for the TTS path without enhancement.

Implement a Source-Aware Filter

For a future-proof fix, modify VADHandler._apply_audio_enhancement to accept a source identifier. Add a flag to VADAudio indicating whether the audio originated from "mic" or "tts", and bypass DFN processing for synthetic audio.

Code Examples

The following examples demonstrate how to configure the pipeline to avoid the conflict and how to implement a conditional enhancement check.

Disabling DFN for Pocket TTS Configurations

Configure the pipeline by explicitly disabling audio enhancement in the VAD arguments while keeping Pocket TTS active:

from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.pocket_tts_arguments import PocketTTSHandlerArguments
from speech_to_speech.arguments_classes.vad_arguments import VADHandlerArguments

vad_args = VADHandlerArguments(
    audio_enhancement=False,   # Critical: disables DFN for TTS compatibility

    # other VAD args …

)

tts_args = PocketTTSHandlerArguments(
    device="cpu",
    voice="jean",
    sample_rate=16000,
)

pipeline = SpeechToSpeechPipeline(
    vad_handler_kwargs=vad_args,
    pocket_tts_handler_kwargs=tts_args,
    # other handlers …

)

pipeline.run()

Conditional Enhancement by Source Type

Modify the enhancement method to skip processing for TTS-generated audio:

def _apply_audio_enhancement(self, array: np.ndarray, source: str = "mic") -> np.ndarray:
    # source = "mic" for microphone, "tts" for synthetic audio

    if source == "tts":
        # Bypass DFN for TTS‑generated audio

        return array
    # existing DFN path …

    enhanced = self.dfn_model(array)
    return enhanced

Summary

• DeepFilterNet audio enhancement in src/speech_to_speech/VAD/vad_handler.py applies noise suppression to final VAD chunks using torchaudio resampling.
• Pocket TTS in src/speech_to_speech/TTS/pocket_tts_handler.py produces pre-resampled audio blocks that conflict with DFN's processing assumptions.
• The audio_enhancement flag is global and does not distinguish between microphone input and TTS output, causing double resampling and potential audio corruption.
• Resolution: Set audio_enhancement=False in VADHandlerArguments when using Pocket TTS, or implement source-aware filtering to bypass enhancement for synthetic audio.

Frequently Asked Questions

Can I use DeepFilterNet and Pocket TTS in the same pipeline instance?

No. According to the huggingface/speech-to-speech source code, these features are mutually exclusive because they operate on the same audio stream with incompatible expectations. DeepFilterNet expects raw microphone input for noise suppression, while Pocket TTS generates pre-resampled synthetic audio that does not require enhancement. Attempting to use both simultaneously causes the VAD handler to apply DFN processing to TTS output, resulting in waveform misalignment.

Why does DeepFilterNet corrupt Pocket TTS audio specifically?

DeepFilterNet uses torchaudio.functional.resample to align sample rates based on self.df_state.sr(). Since Pocket TTS already resamples its output to the pipeline's target sample_rate (e.g., 16 kHz) in pocket_tts_handler.py lines 175–200, the additional resampling step inside _apply_audio_enhancement creates length mismatches. Furthermore, DFN is optimized for natural speech acoustics, not synthetic TTS waveforms, leading to unnecessary computational overhead and potential audio artifacts.

How do I disable audio enhancement only for TTS but keep it for microphone input?

Currently, the repository does not provide a built-in source discriminator in VADAudio objects. As a workaround, you can run separate pipeline instances—one with audio_enhancement=True for STT processing and one with audio_enhancement=False for TTS generation. Alternatively, you can contribute a fix to src/speech_to_speech/VAD/vad_handler.py by adding a source parameter to _apply_audio_enhancement and modifying the VAD logic to tag TTS audio appropriately.

Which configuration files control these settings?

The behavior is controlled through argument classes defined in src/speech_to_speech/arguments_classes/vad_arguments.py (containing the audio_enhancement boolean) and src/speech_to_speech/arguments_classes/pocket_tts_arguments.py (containing sample_rate and blocksize parameters). These are passed to SpeechToSpeechPipeline in src/speech_to_speech/s2s_pipeline.py, which orchestrates the handler initialization and data flow between VAD and TTS components.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}