How Bilistream Parses YouTube Live Stream Data from HTML

Bilistream checks YouTube live status by downloading the channel’s public live page, extracting the ytInitialData JSON embedded in a <script> tag using a regex, and inspecting the nested isLive flag—no YouTube Data API required.

Bilistream is an open‑source streaming bridge that monitors YouTube channels for live broadcasts without relying on official API quotas or authentication keys. Instead, it scrapes the public HTML page at https://www.youtube.com/channel/<ID>/live and parses the embedded state object to determine if a stream is active. This approach keeps the tool lightweight, self‑hosted, and free of external API dependencies.

The HTML Parsing Strategy

The core logic resides in src/plugins/live.rs inside the get_youtube_live_status function. Rather than using the YouTube Data API, the code performs a two‑step extraction:

1. Fetch and prettify the HTML page to ensure consistent formatting for regex matching.
2. Extract and traverse the ytInitialData JSON blob to locate the boolean isLive property deep inside the videoViewCountRenderer object.

This method works because YouTube injects a global JavaScript variable named ytInitialData into every page, containing the full initial state of the rendered content.

Step‑by‑Step Implementation

Building the HTTP Client

In src/plugins/live.rs lines 46‑55, Bilistream constructs a reqwest client with cookie support and transient‑error retry logic:

• Enables the cookie store to handle consent banners or regional redirects.
• Sets a 30‑second timeout to prevent hanging requests.
• Wraps the client with RetryTransientMiddleware for automatic retries on network failures.

// src/plugins/live.rs (lines 46-55)
let client = ClientBuilder::new(
    reqwest::Client::builder()
        .cookie_store(true)
        .timeout(std::time::Duration::new(30, 0))
        .build()?
)
.with(RetryTransientMiddleware::new_with_policy(retry_policy))
.build();

Fetching the Channel Live Page

Line 57 initiates a GET request to the channel’s live endpoint:

// src/plugins/live.rs (line 57)
let url = format!("https://www.youtube.com/channel/{}/live", channel_name);
let resp = client.get(&url).send().await?.text().await?;

Before parsing, line 63 passes the raw HTML through prettyish_html::prettify to normalize whitespace and indentation, making the subsequent regex match more reliable.

Extracting ytInitialData with Regex

Lines 67‑71 use a regex to capture the JSON payload from a <script> tag containing the var ytInitialData = ...; declaration:

// src/plugins/live.rs (lines 67-71)
let re = regex::Regex::new(r#"\s*<script nonce=".*">var ytInitialData = (.*);\s*?</script>"#).unwrap();
let caps = re.captures(&html).ok_or("No ytInitialData found")?;
let json_str = caps.get(1).ok_or("No JSON match")?.as_str();

The captured group contains the raw JSON string, which line 78 deserializes into a serde_json::Value:

// src/plugins/live.rs (line 78)
let data: serde_json::Value = serde_json::from_str(json_str)?;

Navigating the JSON Structure

Lines 79‑82 walk the nested JSON tree to reach the live‑status flag:

// src/plugins/live.rs (lines 79-82)
let is_live = data
    .pointer("/contents/twoColumnWatchNextResults/results/results/contents/0/videoPrimaryInfoRenderer/viewCount/videoViewCountRenderer/isLive")
    .and_then(|v| v.as_str())
    .unwrap_or("false");

The code specifically checks the field: contents.twoColumnWatchNextResults.results.results.contents[0].videoPrimaryInfoRenderer.viewCount.videoViewCountRenderer.isLive.

Finally, lines 86‑90 return a boolean result:

// src/plugins/live.rs (lines 86-90)
match is_live {
    "true" => Ok(true),
    _ => Ok(false),
}

Integration with the Youtube Plugin

The high‑level Youtube plugin in src/plugins/youtube.rs delegates its status check to the helper above. At line 36, the get_status method simply forwards the call:

// src/plugins/youtube.rs (line 36)
async fn get_status(&self) -> Result<bool, Box<dyn Error>> {
    get_youtube_live_status(&self.room).await
}

This separation keeps the HTML‑parsing logic isolated in live.rs while the Youtube struct manages configuration and middleware.

Usage Examples

Directly Checking a Channel

You can invoke the internal helper to query any public channel ID:

use bilistream::plugins::live::get_youtube_live_status;
use std::error::Error;
use tokio;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let channel_id = "UCSJ4gkVC6NrvII8umztf0Ow"; // example: Lofi Girl
    let is_live = get_youtube_live_status(channel_id).await?;
    println!("Channel {} is live: {}", channel_id, is_live);
    Ok(())
}

Using the Youtube Plugin

For production use, instantiate the plugin with a configured client:

use bilistream::plugins::youtube::Youtube;
use bilistream::config::Config;
use reqwest_middleware::ClientBuilder;
use std::error::Error;
use tokio;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let config = Config::default();
    
    let raw = reqwest::Client::builder()
        .cookie_store(true)
        .timeout(std::time::Duration::new(30, 0))
        .build()?;
    let client = ClientBuilder::new(raw).build();
    
    let yt = Youtube::new("UCSJ4gkVC6NrvII8umztf0Ow", String::new(), client, config);
    let live = yt.get_status().await?;
    println!("Live? {}", live);
    Ok(())
}

Summary

• No API key required – Bilistream scrapes the public /live page instead of using the YouTube Data API.
• Regex extraction – The ytInitialData JSON is captured from a <script> tag using the pattern \s*<script nonce=".*">var ytInitialData = (.*);\s*?</script> in src/plugins/live.rs.
• Deep JSON traversal – The code walks to contents.twoColumnWatchNextResults.results.results.contents[0].videoPrimaryInfoRenderer.viewCount.videoViewCountRenderer.isLive to read the live flag.
• Robust networking – The client includes cookie support, a 30‑second timeout, and automatic retry middleware.
• Modular design – The Youtube plugin delegates to get_youtube_live_status, keeping parsing logic separate from plugin orchestration.

Frequently Asked Questions

Does Bilistream require a YouTube Data API key?

No. Bilistream parses the public HTML page directly, extracting the embedded ytInitialData variable. This eliminates API quota concerns and authentication complexity, though it depends on the stability of YouTube’s internal page structure.

What happens if YouTube changes their HTML structure?

If YouTube modifies the ytInitialData schema or moves the isLive field, the JSON pointer in src/plugins/live.rs lines 79‑82 would need updating. The regex itself is tolerant of nonce attribute changes but assumes the variable name remains ytInitialData.

How does Bilistream handle network interruptions?

The reqwest client is wrapped with RetryTransientMiddleware (configured in lines 46‑55), which automatically retries transient HTTP errors such as timeouts or temporary 5xx responses before failing.

Can this method detect scheduled streams or premieres?

The current implementation checks the isLive flag inside videoViewCountRenderer, which only returns "true" during an active broadcast. Scheduled streams that have not started will typically show "false" or omit the field entirely, meaning Bilistream treats them as offline until the stream actually begins.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}