Data Flow from transcribe.py to pack_transcripts.py to takes_packed.md for LLM Transcript Reading

The pipeline extracts audio from video files into word-level JSON transcripts using transcribe.py, aggregates those words into phrase-level segments via pack_transcripts.py based on silence thresholds and speaker changes, and outputs a compact takes_packed.md file that serves as the primary reading material for LLMs.

The browser-use/video-use repository implements a three-stage pipeline that transforms raw video into a structured, LLM-optimized markdown format. This data flow from transcribe.py to pack_transcripts.py to takes_packed.md ensures that language models receive high-resolution temporal cues while minimizing token count by grouping words into meaningful phrases.

Stage 1: Audio Extraction and JSON Transcription

The process begins in helpers/transcribe.py, which handles audio preprocessing and API communication with ElevenLabs Scribe.

Extracting Mono 16 kHz Audio

The script calls extract_audio to generate a mono 16 kHz WAV file from the source video, optimizing the format for speech recognition APIs.

Caching Word-Level JSON Output

After uploading to ElevenLabs Scribe, the API returns a JSON payload containing word-level timestamps, speaker IDs, and audio events. According to the source code at lines 23-27, this response is written to <edit_dir>/transcripts/<video-stem>.json and cached for subsequent runs, preventing redundant API calls.

Stage 2: Phrase-Level Aggregation

The aggregation logic resides in helpers/pack_transcripts.py, which transforms granular word data into coherent phrases suitable for LLM consumption.

Scanning Transcript Directories

The packer initializes by scanning the transcripts directory. As shown at lines 87-89, the code uses json_files = sorted(transcripts_dir.glob("*.json")) to collect all cached transcription files for processing.

Grouping Words into Phrases

The core logic relies on the group_into_phrases function, which iterates through the "words" array from each JSON file. The function implements a flush mechanism (lines 55-81) that aggregates words into phrase dictionaries containing start, end, text, and speaker_id fields.

Handling Speaker Changes and Silence Thresholds

Phrases are flushed based on two specific conditions documented in the source:

• Silence detection: When a spacing entry (type == "spacing") exceeds the default threshold of 0.5 seconds (lines 90-98)
• Speaker transitions: When the speaker_id changes between consecutive words (lines 107-110)

Stage 3: Markdown Generation for LLM Consumption

The final output stage converts the structured phrase data into a human-readable, machine-parseable format.

Time-Annotated Format

The render_markdown function generates lines using the format_time utility to create precise time ranges. Each line follows the pattern documented at lines 44-61:

[002.52-005.36] S0 Ninety percent of what a web agent does is completely wasted.

The optional speaker tag (S0, S1, etc.) prefixes the text content when speaker identification is available.

The LLM Reading View

As documented in SKILL.md (lines 12-14), the resulting takes_packed.md file serves as the LLM's primary reading view. The file is written to <edit_dir>/takes_packed.md (lines 94-95 in pack_transcripts.py), providing a compact representation that preserves temporal context while reducing token overhead compared to raw word-level data.

Running the Pipeline

Execute the scripts sequentially to process video content.

Transcribe a single video to create the JSON cache:

python helpers/transcribe.py path/to/video.mp4 --edit-dir ./my_edit

Pack all transcripts into the LLM-readable markdown:

python helpers/pack_transcripts.py --edit-dir ./my_edit

The generated takes_packed.md organizes content by video segments with phrase-level granularity:

## C0103  (duration: 43.0s, 8 phrases)

  [002.52-005.36] S0 Ninety percent of what a web agent does is completely wasted.
  [006.08-006.74] S0 We fixed this.

Summary

• helpers/transcribe.py extracts mono 16 kHz audio and caches word-level JSON transcripts from ElevenLabs Scribe to <edit_dir>/transcripts/<video-stem>.json
• helpers/pack_transcripts.py scans all JSON files, uses group_into_phrases to aggregate words based on 0.5-second silence thresholds and speaker changes, and stores phrase data with precise timestamps
• takes_packed.md is generated as the final artifact, formatted with time ranges and speaker IDs, serving as the optimized reading source for LLMs according to the repository's SKILL.md documentation

Frequently Asked Questions

What triggers a new phrase boundary in the packing algorithm?

The group_into_phrases function in pack_transcripts.py flushes the current phrase when either a silence entry exceeds 0.5 seconds or when the speaker_id changes between words, ensuring phrases align with natural speech patterns and speaker turns.

Why does the pipeline use a two-step process instead of direct markdown generation?

The intermediate JSON caching in transcribe.py allows for incremental processing and API cost optimization, while the separate packing stage enables algorithmic refinement of phrase boundaries without re-querying the transcription API.

How does the markdown format preserve temporal information for LLMs?

Each line in takes_packed.md includes a formatted time range (e.g., [002.52-005.36]) generated by the format_time function, allowing LLMs to reference specific moments in the video while reading compact phrase-level text rather than individual words.

Where is the final markdown file located and how is it referenced?

The file is written to <edit_dir>/takes_packed.md by default, as implemented at lines 94-95 in pack_transcripts.py, and is explicitly documented in SKILL.md as the primary source document for LLM-driven transcript reading tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}