# DeepFilterNet Audio Enhancement Conflicts with Pocket TTS in Speech-to-Speech Pipeline

> Resolve DeepFilterNet audio enhancement conflicts with Pocket TTS in your speech-to-speech pipeline. Learn why these components clash and how to fix them for seamless audio processing.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: deep-dive
- Published: 2026-08-03

---

**DeepFilterNet audio enhancement and Pocket TTS are mutually exclusive in the huggingface/speech-to-speech pipeline because they apply incompatible resampling and processing expectations to the same audio stream.**

The huggingface/speech-to-speech repository provides a modular pipeline for real-time voice conversations, optionally integrating DeepFilterNet (DFN) for microphone noise suppression and Pocket TTS for streaming text-to-speech synthesis. However, enabling both features simultaneously creates a technical conflict where the VAD handler attempts to enhance synthetic TTS audio as if it were raw microphone input. Understanding this DeepFilterNet audio enhancement conflict with Pocket TTS is essential for configuring stable production deployments.

## Why DeepFilterNet and Pocket TTS Conflict

The pipeline treats both microphone input and TTS-generated audio as identical `VADAudio` objects, causing the Voice Activity Detection (VAD) handler to apply DeepFilterNet enhancement indiscriminately.

### How DeepFilterNet Processes Audio

In [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py), the optional audio enhancement is wired through the `audio_enhancement` boolean flag (lines 70–74). When enabled, the `_apply_audio_enhancement` method (lines 700–702) receives the final VAD-emitted audio chunk as a continuous NumPy waveform.

The DFN model expects a single, continuous speech recording and internally uses `torchaudio.functional.resample` to align the sample rate with `self.df_state.sr()`. This processing assumes the input is genuine microphone capture requiring noise suppression, not synthetic speech.

### How Pocket TTS Handles Audio

The `PocketTTSHandler` in [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py) (lines 70–87 and 175–200) generates audio in small blocks (approximately 10 ms) that are pre-resampled to the pipeline's target `sample_rate` (default 16 kHz) before yielding. The `setup` method accepts parameters including `sample_rate: int = 16000` and `blocksize: int = 512`, creating a streaming output that is already optimized for playback.

### The Root Cause of the Incompatibility

The conflict arises because `audio_enhancement` is a global VAD configuration that does not discriminate between audio sources. When Pocket TTS is active, the VAD handler still invokes `_apply_audio_enhancement` on the final chunk containing TTS-generated audio.

This creates two problems:

1. **Domain mismatch**: DeepFilterNet is trained for natural speech recorded on microphones, not synthetic TTS output, leading to potential shape mismatches or unnecessary computation.
2. **Double resampling**: Since Pocket TTS already resamples audio to 16 kHz (or the configured `sample_rate`), the additional `torchaudio` resample step inside DFN misaligns the waveform length and corrupts the audio stream.

## How to Resolve the Conflict

You must ensure that DeepFilterNet enhancement runs only on microphone input, not on TTS-generated streams.

### Disable Audio Enhancement When Using Pocket TTS

The simplest solution is to set `audio_enhancement=False` in your `VADHandlerArguments` when instantiating the pipeline with Pocket TTS enabled.

### Use Separate Pipelines

If your application requires both features in the same session, run two distinct pipeline instances: one for the STT (speech-to-text) path with DFN enabled for microphone cleaning, and a separate one for the TTS path without enhancement.

### Implement a Source-Aware Filter

For a future-proof fix, modify `VADHandler._apply_audio_enhancement` to accept a source identifier. Add a flag to `VADAudio` indicating whether the audio originated from `"mic"` or `"tts"`, and bypass DFN processing for synthetic audio.

## Code Examples

The following examples demonstrate how to configure the pipeline to avoid the conflict and how to implement a conditional enhancement check.

### Disabling DFN for Pocket TTS Configurations

Configure the pipeline by explicitly disabling audio enhancement in the VAD arguments while keeping Pocket TTS active:

```python
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.pocket_tts_arguments import PocketTTSHandlerArguments
from speech_to_speech.arguments_classes.vad_arguments import VADHandlerArguments

vad_args = VADHandlerArguments(
    audio_enhancement=False,   # Critical: disables DFN for TTS compatibility

    # other VAD args …

)

tts_args = PocketTTSHandlerArguments(
    device="cpu",
    voice="jean",
    sample_rate=16000,
)

pipeline = SpeechToSpeechPipeline(
    vad_handler_kwargs=vad_args,
    pocket_tts_handler_kwargs=tts_args,
    # other handlers …

)

pipeline.run()

```

### Conditional Enhancement by Source Type

Modify the enhancement method to skip processing for TTS-generated audio:

```python
def _apply_audio_enhancement(self, array: np.ndarray, source: str = "mic") -> np.ndarray:
    # source = "mic" for microphone, "tts" for synthetic audio

    if source == "tts":
        # Bypass DFN for TTS‑generated audio

        return array
    # existing DFN path …

    enhanced = self.dfn_model(array)
    return enhanced

```

## Summary

- **DeepFilterNet audio enhancement** in [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) applies noise suppression to final VAD chunks using `torchaudio` resampling.
- **Pocket TTS** in [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py) produces pre-resampled audio blocks that conflict with DFN's processing assumptions.
- The `audio_enhancement` flag is global and does not distinguish between microphone input and TTS output, causing double resampling and potential audio corruption.
- **Resolution**: Set `audio_enhancement=False` in `VADHandlerArguments` when using Pocket TTS, or implement source-aware filtering to bypass enhancement for synthetic audio.

## Frequently Asked Questions

### Can I use DeepFilterNet and Pocket TTS in the same pipeline instance?

No. According to the huggingface/speech-to-speech source code, these features are mutually exclusive because they operate on the same audio stream with incompatible expectations. DeepFilterNet expects raw microphone input for noise suppression, while Pocket TTS generates pre-resampled synthetic audio that does not require enhancement. Attempting to use both simultaneously causes the VAD handler to apply DFN processing to TTS output, resulting in waveform misalignment.

### Why does DeepFilterNet corrupt Pocket TTS audio specifically?

DeepFilterNet uses `torchaudio.functional.resample` to align sample rates based on `self.df_state.sr()`. Since Pocket TTS already resamples its output to the pipeline's target `sample_rate` (e.g., 16 kHz) in [`pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/pocket_tts_handler.py) lines 175–200, the additional resampling step inside `_apply_audio_enhancement` creates length mismatches. Furthermore, DFN is optimized for natural speech acoustics, not synthetic TTS waveforms, leading to unnecessary computational overhead and potential audio artifacts.

### How do I disable audio enhancement only for TTS but keep it for microphone input?

Currently, the repository does not provide a built-in source discriminator in `VADAudio` objects. As a workaround, you can run separate pipeline instances—one with `audio_enhancement=True` for STT processing and one with `audio_enhancement=False` for TTS generation. Alternatively, you can contribute a fix to [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) by adding a `source` parameter to `_apply_audio_enhancement` and modifying the VAD logic to tag TTS audio appropriately.

### Which configuration files control these settings?

The behavior is controlled through argument classes defined in [`src/speech_to_speech/arguments_classes/vad_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/vad_arguments.py) (containing the `audio_enhancement` boolean) and [`src/speech_to_speech/arguments_classes/pocket_tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/pocket_tts_arguments.py) (containing `sample_rate` and `blocksize` parameters). These are passed to `SpeechToSpeechPipeline` in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py), which orchestrates the handler initialization and data flow between VAD and TTS components.