How OpenScreen Uses PixiJS for Video Playback, Compositing, and Rendering Effects

OpenScreen leverages PixiJS as its high-performance 2D rendering engine to composite video layers, apply real-time filters like Gaussian and motion blur, and manage camera transforms through a hierarchical scene graph of Containers and Sprites.

OpenScreen, an open-source video editing application by siddharthvaddem, utilizes PixiJS to handle complex video playback compositing and rendering effects. The library provides the WebGL-backed graphics pipeline that powers both the interactive preview interface and the high-quality export renderer. By wrapping HTML video elements in PixiJS textures and organizing visual layers through a scene graph, OpenScreen achieves cinematic effects including motion blur, dynamic zoom, and masking while maintaining responsive UI performance.

Core PixiJS Architecture in OpenScreen

OpenScreen builds its rendering pipeline around several key PixiJS classes that manage the graphics context, scene hierarchy, and visual assets.

Application and Rendering Context

The Application class serves as the entry point for PixiJS in OpenScreen, creating the canvas and WebGL rendering context. The codebase initializes separate instances for the playback UI and the export pipeline. In src/components/video-editor/VideoPlayback.tsx, the application is instantiated within a React useEffect hook, while src/lib/exporter/frameRenderer.ts creates an off-screen instance for frame-by-frame video export. According to the source code at lines 40-52 of frameRenderer.ts, the export renderer configures the application with specific dimensions and resolution settings optimized for high-quality output.

Scene Graph Hierarchy

OpenScreen organizes visual elements using PixiJS Container objects to create a logical layer hierarchy. The architecture defines a cameraContainer as the top-level group that holds the video sprite, webcam overlay, and masking graphics. Nested within is the videoContainer, which contains the raw video sprite and its associated mask. As implemented in VideoPlayback.tsx at lines 94-100, this nesting allows OpenScreen to apply transforms—such as zoom and pan operations—to the entire camera view in a single step by modifying the parent container.

Video Compositing and Layer Management

The compositing system relies on PixiJS Sprite and Graphics classes to blend video streams with visual effects and masks.

Video Sprites and Textures

OpenScreen converts HTML5 <video> elements into PixiJS textures to enable GPU-accelerated rendering. The process uses Texture.from(videoElement) to create a live texture reference, which is then wrapped in a Sprite object. According to VideoPlayback.tsx at lines 122-130, this sprite is inserted into the videoContainer, allowing PixiJS to treat the video frame as a standard texture that can be manipulated with filters and transforms alongside other graphical elements.

Masking with Graphics

To clip video content to specific shapes—such as rounded rectangles for webcam overlays—OpenScreen utilizes the Graphics class. The system draws vector shapes using Graphics methods, then assigns the instance to videoContainer.mask to enforce clipping at render time. The frameRenderer.ts file at lines 95-100 demonstrates this pattern, where mask graphics are configured to define visible regions for layered video elements.

Real-Time Rendering Effects

OpenScreen implements cinematic visual effects through PixiJS's filter system, specifically leveraging the pixi-filters library for advanced blur effects.

Filter Pipeline Configuration

The application attaches filters directly to container objects to affect all child elements. The videoContainer typically holds an array of filters including BlurFilter for static Gaussian blur and MotionBlurFilter for directional blur effects. As shown in frameRenderer.ts at lines 61-68, these filters are instantiated during initialization and assigned to the container's filters property, enabling real-time post-processing with minimal performance overhead.

Motion Blur Dynamics

The motion blur implementation tracks camera velocity to generate cinematic "speed" effects. The applyZoomTransform function in src/components/video-editor/videoPlayback/zoomTransform.ts (lines 78-90) calculates movement delta between frames and feeds a velocity vector to MotionBlurFilter. The filter's kernel size and offset are dynamically tuned based on camera speed, creating a smooth blur that naturally fades when movement stops. This state management uses a MotionBlurState object to track previous positions and scale values across animation frames.

Camera Transforms and Zoom

OpenScreen handles camera movements through mathematical transforms applied to the PixiJS scene graph.

Zoom Transform Calculations

The computeZoomTransform function calculates scale and translation values based on stage dimensions, base mask coordinates, zoom scale factors, and focus points. This utility—located in zoomTransform.ts—returns transform parameters that are applied directly to the cameraContainer object's scale and position properties. When users interact with the preview to set focus points, computeFocusFromTransform reverses these calculations to synchronize UI overlays with the video's logical coordinate space.

Export Pipeline and Off-Screen Rendering

For video export, OpenScreen utilizes a dedicated rendering path that mirrors the preview architecture while optimizing for frame-by-frame capture.

The FrameRenderer class in frameRenderer.ts initializes a separate PixiJS Application configured for off-screen rendering. This renderer composites the video layer, applies blur and motion blur filters, and draws the result to a dedicated canvas. After PixiJS renders the stage, the system composites additional layers—such as shadows drawn to a separate shadowCanvas and background wallpaper textures—using standard 2D Canvas APIs before encoding the final frame. This separation keeps the real-time preview responsive while ensuring export quality.

Implementation Examples

Initializing PixiJS with Video and Blur Effects

The following pattern from OpenScreen demonstrates creating a PixiJS application, converting a video element to a sprite, and applying a Gaussian blur filter:

import { Application, Sprite, BlurFilter, Texture } from 'pixi.js';

async function initPixi(videoEl: HTMLVideoElement, canvas: HTMLCanvasElement) {
  const app = new Application({
    view: canvas,
    width: 1280,
    height: 720,
    backgroundAlpha: 0,
    antialias: true,
  });

  // Convert HTML video to PixiJS texture and sprite
  const tex = await Texture.from(videoEl);
  const videoSprite = new Sprite(tex);
  app.stage.addChild(videoSprite);

  // Apply static Gaussian blur
  const blur = new BlurFilter();
  blur.blur = 5;
  videoSprite.filters = [blur];

  return { app, videoSprite };
}

This initialization matches the approach used in VideoPlayback.tsx, where Texture.from() bridges HTML media elements with the WebGL graphics pipeline.

Applying Zoom Transforms with Motion Blur

OpenScreen encapsulates camera movement and motion blur updates in the applyZoomTransform utility:

import {
  applyZoomTransform,
  computeZoomTransform,
  createMotionBlurState,
} from '@/components/video-editor/videoPlayback/zoomTransform';
import { MotionBlurFilter } from 'pixi-filters/motion-blur';
import { Container, BlurFilter } from 'pixi.js';

function updateCamera(
  camContainer: Container,
  blur: BlurFilter,
  motionBlur: MotionBlurFilter,
  opts: {
    stageSize: { w: number; h: number };
    baseMask: { x: number; y: number; w: number; h: number };
    zoomScale: number;
    focusX: number;
    focusY: number;
    isPlaying: boolean;
    motionBlurAmount: number;
    motionBlurState: ReturnType<typeof createMotionBlurState>;
  }
) {
  applyZoomTransform({
    cameraContainer: camContainer,
    blurFilter: blur,
    motionBlurFilter: motionBlur,
    stageSize: opts.stageSize,
    baseMask: opts.baseMask,
    zoomScale: opts.zoomScale,
    focusX: opts.focusX,
    focusY: opts.focusY,
    motionIntensity: 0,
    motionVector: undefined,
    isPlaying: opts.isPlaying,
    motionBlurAmount: opts.motionBlurAmount,
    motionBlurState: opts.motionBlurState,
  });
}

This pattern demonstrates how OpenScreen combines geometric transforms with filter parameter updates, mutating the MotionBlurFilter velocity property in-place based on calculated camera movement.

Summary

• OpenScreen uses PixiJS Application to create separate rendering contexts for the interactive UI (VideoPlayback.tsx) and the export pipeline (frameRenderer.ts).
• The scene graph organizes video layers using nested Container objects (cameraContainer and videoContainer) to group transforms and filters efficiently.
• Video elements are rendered as Sprite objects using Texture.from(videoElement), enabling GPU-accelerated manipulation of HTML5 video streams.
• Masking is implemented via the Graphics class assigned to container mask properties, clipping video to custom shapes at render time.
• Visual effects utilize BlurFilter and MotionBlurFilter from the pixi-filters library, with motion blur velocity calculated dynamically based on camera movement in zoomTransform.ts.
• The export system composites PixiJS-rendered video layers with additional Canvas 2D elements (shadows, backgrounds) to produce final output frames.

Frequently Asked Questions

How does OpenScreen convert HTML5 video elements for PixiJS rendering?

OpenScreen uses Texture.from(videoElement) to create a live texture reference from the HTML5 <video> element, then wraps it in a Sprite instance. As implemented in VideoPlayback.tsx at lines 122-130, this allows PixiJS to treat the video stream as a standard texture that can be positioned, scaled, and filtered alongside other graphical elements in the WebGL pipeline.

What PixiJS filters does OpenScreen use for rendering effects?

OpenScreen primarily utilizes BlurFilter for static Gaussian blur and MotionBlurFilter (from the pixi-filters package) for directional blur effects. According to frameRenderer.ts at lines 61-68, these filters are attached to the videoContainer object, affecting all child elements including the video sprite and mask layers.

How is camera zoom implemented in the PixiJS scene graph?

Camera zoom is implemented by calculating transform values in computeZoomTransform and applying them to the cameraContainer object's scale and position properties. The applyZoomTransform function in zoomTransform.ts handles both the geometric transformation and the updating of motion blur velocity based on camera movement delta, creating smooth animated zoom effects.

Why does OpenScreen use separate PixiJS instances for preview and export?

OpenScreen maintains separate Application instances to optimize for different use cases: the preview instance in VideoPlayback.tsx prioritizes real-time responsiveness and user interaction, while the export instance in frameRenderer.ts is configured for off-screen rendering and frame-by-frame capture. This separation allows the export pipeline to composite additional layers (shadows, backgrounds) using standard Canvas 2D APIs after PixiJS renders the video frame, without impacting the performance of the interactive preview interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}