How the Free-TV/IPTV Script Handles Duplicate Channel Entries Across Different Sources

The playlist generator does not perform deduplication; it writes every marked channel entry directly to the output files, leaving duplicate prevention to manual contributor curation.

The Free-TV/IPTV repository aggregates television channels from multiple Markdown source files into standardized M3U8 playlists. Understanding how the generation script processes duplicate channel entries is crucial for maintainers and contributors who want to avoid redundant listings in the final output.

The Playlist Generation Pipeline

The make_playlist.py script processes source files through a linear pipeline that preserves every valid entry it encounters.

Source File Collection

The script begins by enumerating all Markdown files in the lists/ directory. It uses os.listdir(lists_dir) to collect source files (line 38) and iterates through each file to extract channel data.

Channel Parsing and Instantiation

During processing, the script filters for lines containing the [>] marker, which indicates active channel entries. The parsing loop explicitly skips lines without this marker using if "[>]" not in line: continue (line 53). For each valid line, it instantiates a Channel object that extracts the channel number, name, URL, logo, and metadata (lines 94-105).

Direct Output Without Deduplication

Immediately after creating the Channel object, the script formats and writes the M3U line using print(m3u_line, file=playlist) (lines 56-58). This operation occurs for both the master playlist and country-specific playlists without any intermediate registry or duplicate detection mechanism.

Why Duplicate Channel Entries Are Preserved in the Output

The script maintains no runtime registry of URLs or channel names. Because make_playlist.py does not compare newly parsed entries against previously written ones, identical channels appearing in multiple source files (e.g., usa.md and canada.md) result in multiple identical #EXTINF entries in the final playlist.m3u8.

Manual Prevention of Duplicate Channel Entries

Since the codebase lacks automated duplicate filtering, the repository relies on contribution guidelines to enforce uniqueness. Contributors must ensure that any given channel URL appears in only one country list, preventing the script from generating redundant entries during the build process.

# Simplified excerpt from make_playlist.py

for line in markup_file:
    if "[>]" not in line:
        continue                       # only include lines marked for inclusion

    channel = Channel(group, line, country_code)
    m3u_line = channel.to_m3u_line()
    print(m3u_line, file=playlist)    # writes directly, no duplicate check

If two different Markdown files both contain the same channel line, the resulting playlist will contain duplicate entries because the script does not filter them out.

Summary

• The make_playlist.py script processes every marked channel line without checking for existing entries.
• Duplicate channel entries are preserved in the output M3U8 files when the same URL appears in multiple source Markdown files.
• No automated deduplication registry exists in the playlist generation workflow.
• Contributors must manually prevent duplicates by ensuring each channel appears in only one lists/*.md file.

Frequently Asked Questions

Does the IPTV script remove duplicate channels automatically?

No, the script does not remove duplicates. It writes every channel entry containing the [>] marker directly to the output playlists without comparing against previously processed entries. Deduplication is not implemented in the generation logic.

What happens if the same URL appears in two different country files?

The script will generate duplicate entries in the final playlist.m3u8. If both usa.md and canada.md contain the same channel line, the resulting playlist will include two identical #EXTINF entries because the script processes each file independently without cross-referencing.

How can contributors prevent duplicate channel entries?

Contributors must ensure each channel URL appears in only one country-specific Markdown file within the lists/ directory. The repository's contribution guidelines require manual verification to avoid adding channels that already exist in other source files.

Which part of the code is responsible for deduplication?

There is no deduplication code in the current implementation. The make_playlist.py script (specifically lines 56-58) writes channels to the playlist file immediately after parsing, without any conditional checks for existing URLs or names.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}