How the Log Parser Extracts Stream Information in the IPTV-Org Repository

The LogParser class reads the generators.log file line-by-line, parses each JSON entry into a LogItem object, and returns a typed array containing stream metadata including file type, path, and channel count.

The iptv-org/iptv repository manages thousands of M3U playlist files organized by categories, countries, languages, and regions. To track these generation artifacts, the build pipeline writes structured logs that the log parser extracts stream information from, enabling automated reporting and table generation. Understanding this parsing mechanism reveals how the repository maintains its comprehensive index of IPTV streams.

Understanding the generators.log File Structure

Before parsing occurs, the generation workflow writes a plain-text log file at logs/generators.log. Each line represents a single artifact produced during the build process:

{"type":"region","filepath":"regions/afr.m3u","count":2}
{"type":"category","filepath":"categories/entertainment.m3u","count":150}
{"type":"country","filepath":"countries/us.m3u","count":450}

The LogItem type definition in scripts/core/logParser.ts formalizes this structure:

export type LogItem = {
  type: string
  filepath: string
  count: number
}

Core Log Parser Implementation

The LogParser Class

Located at scripts/core/logParser.ts, the LogParser class provides a single method that transforms raw log content into typed objects. The implementation handles edge cases like empty content and blank lines:

export class LogParser {
  parse(content: string): LogItem[] {
    if (!content) return []
    const lines = content.split('\n')
    return lines.map(line => (line ? JSON.parse(line) : null)).filter(l => l)
  }
}

The parsing logic follows a strict pipeline:

1. Guard clause returns empty array for falsy input
2. Line splitting breaks the file on newline characters
3. JSON parsing converts each non-empty line to an object
4. Null filtering removes empty entries resulting from blank lines

How Downstream Components Use Parsed Stream Data

The parsed LogItem array powers multiple table generators across the repository. Each component filters the logs by type to extract relevant statistics.

Region Statistics Generation

In scripts/tables/regionsTable.ts, the parser collects regional playlist data:

const parser = new LogParser()
const logsStorage = new Storage(LOGS_DIR)
const generatorsLog = await logsStorage.load('generators.log')
const parsed = parser.parse(generatorsLog)
const logRegions = parsed.filter((logItem: LogItem) => logItem.type === 'region')

Language and Category Aggregation

Similar patterns appear in scripts/tables/languagesTable.ts and scripts/tables/categoriesTable.ts:

const parser = new LogParser()
const generatorsLog = await logsStorage.load('generators.log')
const parsed = parser.parse(generatorsLog)
const logLanguages = parsed.filter(l => l.type === 'language')

Step-by-Step Stream Information Extraction Process

When the build pipeline needs to analyze generated playlists, it follows this exact sequence:

1. Load raw content using Storage(LOGS_DIR).load('generators.log') to read the entire file as a string
2. Instantiate parser by creating new LogParser() (the class maintains no internal state)
3. Split into lines via content.split('\n') to process each entry individually
4. Parse JSON entries using JSON.parse(line) for every non-empty line
5. Filter valid objects to remove null values from blank lines or parsing errors
6. Return typed array of LogItem objects ready for filtering and aggregation

Practical Example: Querying Stream Statistics

The following script demonstrates how to extract comprehensive statistics from the log file using the LogParser:

import { Storage } from '@freearhey/storage-js'
import { LogParser, LogItem } from '../core'
import { LOGS_DIR } from '../constants'

async function getStreamStats() {
  // Load the raw log content
  const logsStorage = new Storage(LOGS_DIR)
  const rawLog = await logsStorage.load('generators.log')

  // Parse into structured objects
  const parser = new LogParser()
  const items: LogItem[] = parser.parse(rawLog)

  // Aggregate total stream count per file type
  const stats = items.reduce((acc, cur) => {
    acc[cur.type] = (acc[cur.type] ?? 0) + cur.count
    return acc
  }, {} as Record<string, number>)

  console.log('Stream statistics by log type:', stats)
}

getStreamStats()

Executing this script produces aggregated output similar to:

Stream statistics by log type: {
  raw: 14,
  category: 31,
  language: 5,
  country: 4,
  region: 23,
  source: 8,
  index: 4
}

Summary

• The LogParser class in scripts/core/logParser.ts converts raw generators.log text into typed LogItem objects
• Each log entry contains three fields: type (artifact category), filepath (relative path), and count (number of streams)
• The parser handles edge cases by filtering empty lines and returning an empty array for null input
• Downstream table generators in scripts/tables/ use the parsed data to create markdown statistics for regions, languages, and categories
• The implementation relies on standard JavaScript methods: split('\n'), JSON.parse(), and filter()

Frequently Asked Questions

What file format does the generators.log use?

The generators.log file uses a newline-delimited JSON (NDJSON) format. Each line contains a single JSON object representing one generated artifact, with no outer array or commas between entries. This format allows the LogParser to process the file line-by-line using split('\n') and JSON.parse().

How does the LogParser handle malformed or empty lines?

The parser implements defensive programming to handle irregular input. When splitting content by newlines, it maps each line through a conditional: line ? JSON.parse(line) : null. Empty strings become null values, which are then removed by the final .filter(l => l) operation. If the input content is falsy, the method returns an empty array immediately.

Can I use LogParser to analyze custom log files?

Yes, the LogParser is generic and stateless. As long as your log file contains newline-separated JSON objects with type, filepath, and count fields (or any structure you define), you can instantiate new LogParser() and call parse(content) to receive a typed array. The class has no dependencies on specific file paths or storage implementations.

Where does the generators.log file get created?

The generators.log file is written during the playlist generation workflow, specifically in scripts/commands/playlist/generate.ts. As the command generates raw playlists, category files, regional collections, and other artifacts, it appends a JSON line to the log for each file created. The Storage class from @freearhey/storage-js handles the file I/O operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}