How Screencast and Video Recording Works in Ego Lite: CDP and FFmpeg Explained

The Ego Lite browser harness captures screen recordings by streaming raw JPEG frames over Chrome DevTools Protocol (CDP) and piping them through FFmpeg for VP8 encoding into WebM files.

The screencast system in ego-lite provides automated agents with a lightweight, disposable API for recording browser interactions. Built on TypeScript and FFmpeg, it avoids heavy native dependencies while producing standard WebM output suitable for debugging, demos, and regression testing.

---

Overview: Two-Module Architecture

The video recording functionality splits cleanly between orchestration and encoding:

• startScreencast / stopScreencast (src/driver/screencast.ts) — manages CDP session lifecycle, frame subscription, and timing
• VideoRecorder (src/video-recorder.ts) — wraps FFmpeg for JPEG-to-VP8 transcoding

This separation keeps the JavaScript side deterministic while delegating CPU-intensive encoding to a battle-tested external process.

---

The Driver Layer: Capturing Frames via CDP

Implementation file: src/driver/screencast.ts (source)

Starting a Screencast

When startScreencast() is invoked, it performs six sequential steps:

1. Validates output path — enforces .webm extension and parses quality settings (default: 90)
2. Ensures CDP session — calls ensureSession() and determines capture size via defaultSize (scales viewport down by default) at lines 58-62
3. Spawns recorder — creates a VideoRecorder instance through dependencies.createRecorder and starts the FFmpeg process at lines 62-66
4. Subscribes to frames — listens for Page.screencastFrame events at lines 76-92
5. Adjusts timestamps — rewrites frame timing to ensure monotonic progression
6. Acknowledges frames — sends Page.screencastFrameAck back to the browser to receive the next frame
7. Starts CDP stream — issues Page.startScreencast command at lines 101-112

The subscription handler at lines 76-92 is performance-critical: each incoming JPEG buffer is immediately forwarded to the recorder with an adjusted timestamp, then acknowledged without awaiting disk I/O.

Stopping a Screencast

The stopScreencast function (lines 29-68) handles cleanup:

1. Unsubscribes from Page.screencastFrame to stop frame ingestion
2. Awaits pending frame writes to complete
3. Fallback screenshot — generates a single PNG if zero frames arrived (prevents empty output files)
4. Sends Page.stopScreencast to terminate the browser stream
5. Finalizes FFmpeg and renames the temporary file to the target path

The async-dispose pattern ensures recordings complete even when scripts exit abruptly.

---

The Recorder Layer: FFmpeg Encoding

Implementation file: src/video-recorder.ts (source)

The VideoRecorder class spans 84 lines and performs three core operations:

1. Process Initialization (start())

Lines 44-82 build and spawn FFmpeg with these arguments:

ffmpeg -f image2pipe -i - -c:v vp8 -r 25 -deadline realtime -b:v 1M -auto-alt-ref 0 output.webm

• -f image2pipe — reads concatenated JPEG images from stdin
• -c:v vp8 — VP8 codec for broad browser compatibility
• -r 25 — hard-coded 25 fps output
• -deadline realtime — prioritizes encoding speed over compression ratio

The temporary output path avoids corrupt files if the process crashes.

2. Frame Writing (writeFrame())

Lines 21-31 handle frame timing logic:

writeFrame(buffer: Buffer, timestamp: number): void {
  const frameNumber = Math.floor(timestamp * 25 / 1000); // Convert ms to frame index at 25fps
  const repeats = Math.max(1, frameNumber - this.lastFrameNumber);
  // Queue buffer for 'repeats' frames to smooth dropped intervals
}

The 25 fps assumption means frame intervals are quantized to 40ms boundaries. If CDP delivers frames irregularly, the recorder duplicates buffers to maintain consistent playback speed.

3. Finalization (stop())

Lines 44-85 perform graceful shutdown:

• Flushes remaining frame repeats to stdin
• Closes stdin pipe (FFmpeg interprets this as EOF)
• Awaits process exit code
• Checks stderr for encoding errors
• Cleans up temporary file on failure, or renames on success

Error handling distinguishes between recoverable (temporary cleanup) and fatal (exception throwing) conditions.

---

Usage: High-Level and Low-Level APIs

Page-Aware Convenience Method

// Start a screencast that records the visible viewport into "demo.webm"
const { dispose } = await page.screencast.start({
  path: "demo.webm",      // must end with .webm
  size: { width: 800, height: 600 }, // optional; defaults to scaled viewport
  quality: 85,            // JPEG quality 0-100 (default 90)
});

// ... interact with the page, click, scroll, etc. ...

// When done, stop and finalize the recording
await dispose();   // equivalent to await page.screencast.stop()

The dispose method is idempotent — safe to call multiple times or leave for finalization hooks.

Direct Driver Access

// Using the low-level API directly (rarely needed)
import { startScreencast, stopScreencast } from "ego-browser/src/driver/screencast.js";

await startScreencast({ path: "out.webm" });
// ... perform actions …
await stopScreencast();   // resolves after FFmpeg finishes writing the file

Low-level access bypasses page object lifecycle management but requires manual session coordination.

---

Configuration Reference

Parameter	Type	Default	Description
path	string	(required)	Output file path; must use .webm extension
size.width	number	scaled viewport	Capture width in pixels
size.height	number	scaled viewport	Capture height in pixels
quality	number	90	JPEG compression quality (0-100)

---

Key Source Files and Tests

File	Purpose
src/driver/screencast.ts	CDP orchestration and public API
src/video-recorder.ts	FFmpeg process management
src/driver/screencast.test.mjs	Unit tests for start/stop logic and error paths
scripts/real-browser-e2e/cases/video-recording.mjs	Full integration example

---

Summary

• CDP integration — Page.screencastFrame events deliver raw JPEG buffers from the browser engine
• FFmpeg encoding — VideoRecorder pipes frames through image2pipe to produce VP8 WebM at 25 fps
• Disposable API — startScreencast returns an async-dispose object for reliable cleanup
• Fault tolerance — fallback screenshot prevents empty outputs; temporary files protect against crashes
• Minimal dependencies — requires only a system FFmpeg installation; no native Node.js modules

---

Frequently Asked Questions

What video format does Ego Lite produce?

Ego Lite outputs VP8-encoded WebM files with silent audio tracks. This format was chosen for universal browser support, reasonable compression, and FFmpeg's mature VP8 encoder. The hard-coded 25 fps and realtime deadline prioritize encoding speed over file size.

Why does the recorder duplicate frames?

The writeFrame() method in src/video-recorder.ts calculates frame numbers based on millisecond timestamps and a fixed 25 fps clock. When CDP delivers frames at irregular intervals, the recorder repeats buffers to fill gaps, producing smooth playback even under load. This trade-off accepts minor quality degradation for timing consistency.

Can I change the frame rate or codec?

Not without modifying src/video-recorder.ts. The 25 fps assumption appears in three places: FFmpeg arguments (-r 25), the timestamp-to-frame conversion (timestamp * 25 / 1000), and frame repeat logic. Codec selection (-c:v vp8) is similarly hard-coded. Pull requests to expose these parameters would need to synchronize timing math across the recorder class.

What happens if no frames arrive before stopping?

The stop handler in src/driver/screencast.ts detects zero-frame recordings and falls back to a single screenshot. This prevents empty WebM files that many players reject. The screenshot uses the same size and quality parameters as the attempted recording, providing diagnostic value even when CDP streaming fails.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}