How you-get Handles m3u8 Playlist Parsing for HLS Streams: A Deep Dive into the Source Code

you-get parses m3u8 playlists by extracting segment URLs from the playlist text, resolving relative URIs against the base URL, and returning a flat list of absolute segment URLs via the general_m3u8_extractor function in src/you_get/common.py.

you-get is a popular command-line utility for downloading media from various websites. When handling HTTP Live Streaming (HLS) content, the tool must parse m3u8 playlist files to identify individual media segments. This article examines how you-get implements m3u8 playlist parsing by analyzing the source code in the soimort/you-get repository.

The Core Parsing Logic in general_m3u8_extractor

The heart of you-get's HLS handling resides in the general_m3u8_extractor function located in src/you_get/common.py. This utility function treats m3u8 playlists as simple text files containing lists of media segment URLs, processing them through a five-step pipeline to produce downloadable URLs.

Fetching and Preparing the Playlist Content

The extraction process begins with an HTTP request to retrieve the raw playlist data. The function calls get_content(url, headers=…), which performs an HTTP GET request and returns the m3u8 file as a raw text string. Once fetched, the content is split into individual lines using .split('\n') to prepare for line-by-line processing.

Filtering Comments and Empty Lines

HLS playlists use lines beginning with # for metadata and comments (such as #EXTM3U or #EXTINF tags). The parser strips whitespace from each line using .strip(), then discards any empty strings or lines starting with the # character. This filtering ensures only actual media segment references remain for URL resolution.

Resolving Absolute and Relative URLs

The final processing stage handles URL resolution, distinguishing between absolute and relative segment paths:

• Absolute URLs: If a line already begins with http, the parser treats it as a complete URL and appends it directly to the result list.
• Relative URIs: For lines containing relative paths (such as segment1.ts or ../data/segment2.ts), the function resolves them against the original playlist URL using urllib.parse.urljoin(url, line). This produces a fully-qualified absolute URL regardless of the relative path format.

The function returns a plain Python list containing all resolved segment URLs in playlist order.

Integration with Site Extractors

The general_m3u8_extractor function serves as a shared utility across multiple site-specific extractors. When a particular video host serves HLS content, its extractor fetches the m3u8 URL and passes it to this helper to retrieve the segment list.

For example, in src/you_get/extractors/longzhu.py, the extractor calls:

urls = general_m3u8_extractor(real_url)

Similarly, src/you_get/extractors/douyutv.py utilizes:

urls = general_m3u8_extractor(m3u8_url)

The video-on-demand extractor in src/you_get/extractors/qie_video.py demonstrates advanced usage by capturing both the segment URLs and duration information:

self.streams[stream_id]['src'], dur = general_m3u8_extractor(self.streams[stream_id]['m3u8_url'])

Command-Line Handling with the --m3u8 Flag

you-get provides a command-line option to handle HLS streams differently through the -m or --m3u8 flag (defined in src/you_get/common.py around line 1670). When this flag is active, you-get bypasses the segment-by-segment download process.

Instead of parsing the playlist and downloading individual segments, the tool passes the raw m3u8 URL directly to an external player such as ffmpeg or mpv. This allows the external program to handle the HLS streaming natively, which is useful for live streams or when you want to watch without saving files.

In this mode, the general_m3u8_extractor function is still invoked, but primarily to display the master playlist URL when m3u8_type='master' is passed to the generic printer, ensuring users can see which stream variant is being accessed.

Practical Code Examples

You can leverage the m3u8 parsing functionality directly in your own Python scripts by importing the common module:

from you_get.common import general_m3u8_extractor

# Parse a master playlist or media playlist

playlist_url = 'https://example.com/hls/master.m3u8'
segment_urls = general_m3u8_extractor(playlist_url)

print(f'Found {len(segment_urls)} segments:')
for seg in segment_urls:
    print(seg)

For command-line usage, stream HLS content directly to your default player without downloading segments to disk:

you-get -m "https://example.com/hls/master.m3u8"

Summary

• you-get handles m3u8 playlist parsing through the general_m3u8_extractor function in src/you_get/common.py.
• The parser fetches the raw playlist text, splits it into lines, and filters out comments (lines starting with #) and empty entries.
• It resolves both absolute URLs and relative URIs using urllib.parse.urljoin to generate fully-qualified segment URLs.
• Multiple site extractors (including longzhu.py, douyutv.py, and qie_video.py) rely on this shared utility for HLS stream handling.
• The --m3u8 or -m CLI flag bypasses segment downloading, passing the raw playlist URL directly to external players like ffmpeg or mpv.

Frequently Asked Questions

What is the main function responsible for m3u8 playlist parsing in you-get?

The general_m3u8_extractor function in src/you_get/common.py serves as the primary utility for m3u8 playlist parsing. It handles the entire workflow from fetching the playlist content to resolving segment URLs and returning them as a Python list.

How does you-get handle relative URLs in m3u8 playlists?

When the parser encounters a line that does not start with http, it treats the path as a relative URI. The function uses urllib.parse.urljoin(url, line) to resolve these paths against the base playlist URL, ensuring that segments are correctly located regardless of whether the playlist uses absolute or relative addressing.

Can I use you-get to stream HLS content without downloading segments?

Yes. By passing the -m or --m3u8 flag, you-get passes the raw m3u8 URL directly to an external player such as ffmpeg or mpv instead of downloading individual segments. This is particularly useful for live streams or when you want to watch content without saving files to disk.

Which extractors in you-get utilize the general_m3u8_extractor function?

Several site-specific extractors rely on this shared utility, including src/you_get/extractors/longzhu.py for Longzhu live streams, src/you_get/extractors/douyutv.py for Douyu live streams, and src/you_get/extractors/qie_video.py for video-on-demand content. Each passes the m3u8 URL to the extractor and receives a list of segment URLs in return.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}