How Annotation Regions in OpenScreen Support Text, Images, and Figure Overlays with Styling

OpenScreen implements a unified AnnotationRegion type system that stores position, timing, and styling data for text, image, and figure overlays, with specialized renderers in annotationRenderer.ts and AnnotationOverlay.tsx handling both canvas export and live UI interactions.

OpenScreen is an open-source video editing platform that enables precise overlay management through its annotation region architecture. Each overlay is stored as an AnnotationRegion object defined in src/components/video-editor/types.ts, specifying content type, visual styling, temporal boundaries, and spatial coordinates. This system supports three distinct overlay types—text, image, and figure—with dedicated rendering pipelines for real-time editing and final video export.

The AnnotationRegion Type System

The foundation of OpenScreen's overlay capability lies in the AnnotationRegion interface. This type encapsulates all metadata required to position and style overlays on the video canvas using percentage-based coordinates.

Each region specifies its position (x and y as percentages of the video dimensions), size (width and height as percentages), z-index for stacking order, and optional start/end timestamps (startMs and endMs) to control temporal visibility.

The type field accepts one of three AnnotationType values, each with corresponding styling data:

• text: Renders styled text using AnnotationTextStyle, which includes color, backgroundColor, fontSize, fontFamily, fontWeight, fontStyle, textDecoration, and textAlign.
• image: Displays an image data-URL (data:image/*) using object-contain scaling within the region bounds, ignoring text styling properties.
• figure: Renders directional arrows (e.g., "up-right") using FigureData, which specifies arrowDirection, color, and strokeWidth.

Rendering Architecture: Three-Layer Pipeline

OpenScreen processes annotation regions through three coordinated layers that ensure consistent rendering across editing and export workflows.

Canvas Export Layer

During video export, src/lib/exporter/annotationRenderer.ts manages the rendering pipeline. The module iterates over annotationRegions, filters entries by the current timestamp, sorts them by zIndex, and dispatches to type-specific render helpers:

• renderText: Draws styled text on a <canvas> element using the region's style fields.
• renderImage: Loads the image data-URL and preserves aspect ratio within the region bounds.
• renderArrow: Retrieves SVG path data from the internal ARROW_PATHS table and strokes the arrow using the region's figureData properties for color and stroke width.

This canvas-based approach ensures overlays appear in the final exported video with exact styling fidelity.

Live Editing Layer

While editing, src/components/video-editor/AnnotationOverlay.tsx provides the interactive interface. The component wraps each region in a draggable, resizable <Rnd> (react-rnd) container that:

• Converts percentage-based coordinates to pixel values for the current viewport.
• Conditionally renders the appropriate inner element based on annotation.type: a <span> for text, <img> for images, or an arrow component imported from src/components/video-editor/ArrowSvgs.tsx.
• Applies CSS styling directly to the rendered element, mapping annotation.style properties to inline styles (e.g., color, fontSize, backgroundColor).

This layer enables direct manipulation of position and size while preserving the styling defined in the region's type system.

Playback Integration Layer

The src/components/video-editor/VideoPlayback.tsx component orchestrates overlay visibility during playback. It receives annotationRegions via props, filters them against the current playback time to determine active overlays, sorts by zIndex, and renders an AnnotationOverlay instance for each visible region.

This layer also handles click-through cycling for overlapping annotations, temporarily boosting the selected annotation's z-index to ensure it remains accessible during editing.

Implementation Examples

Defining Annotation Regions

To create overlays, define objects conforming to the AnnotationRegion interface:

import {
  type AnnotationRegion,
  type AnnotationTextStyle,
  type FigureData,
} from "@/components/video-editor/types";

const textStyle: AnnotationTextStyle = {
  color: "#ffffff",
  backgroundColor: "rgba(0,0,0,0.5)",
  fontSize: 36,
  fontFamily: "Inter",
  fontWeight: "bold",
  fontStyle: "italic",
  textDecoration: "underline",
  textAlign: "center",
};

export const myAnnotations: AnnotationRegion[] = [
  // Text overlay
  {
    id: "anno-1",
    startMs: 2000,
    endMs: 8000,
    type: "text",
    content: "Hello, Open Screen!",
    position: { x: 50, y: 20 },
    size: { width: 40, height: 15 },
    style: textStyle,
    zIndex: 10,
  },

  // Image overlay
  {
    id: "anno-2",
    startMs: 5000,
    endMs: 12000,
    type: "image",
    content: "data:image/png;base64,iVBORw0KG…",
    position: { x: 20, y: 60 },
    size: { width: 30, height: 30 },
    style: textStyle,
    zIndex: 20,
  },

  // Figure (arrow) overlay
  {
    id: "anno-3",
    startMs: 0,
    endMs: 15000,
    type: "figure",
    content: "",
    position: { x: 70, y: 70 },
    size: { width: 20, height: 20 },
    style: textStyle,
    figureData: {
      arrowDirection: "up-right",
      color: "#34B27B",
      strokeWidth: 4,
    } as FigureData,
    zIndex: 30,
  },
];

The AnnotationRegion interface is defined in src/components/video-editor/types.ts.

Integrating with Video Playback

Pass the annotation array to the VideoPlayback component:

import VideoPlayback from "@/components/video-editor/VideoPlayback";

<VideoPlayback
  videoPath="/videos/sample.mp4"
  webcamLayoutPreset="picture-in-picture"
  zoomRegions={[]}
  selectedZoomId={null}
  onSelectZoom={() => {}}
  onZoomFocusChange={() => {}}
  isPlaying={true}
  aspectRatio="16:9"
  annotationRegions={myAnnotations}
  selectedAnnotationId={null}
  onSelectAnnotation={(id) => console.log("Selected", id)}
  onAnnotationPositionChange={(id, pos) => {
    // Update region position in state
  }}
  onAnnotationSizeChange={(id, size) => {
    // Update region size in state
  }}
/>

The annotationRegions prop is consumed in src/components/video-editor/VideoPlayback.tsx to render active overlays.

Applying Styles in Live Overlays

Inside AnnotationOverlay.tsx, text styling is applied via inline CSS:

<span
  style={{
    color: annotation.style.color,
    backgroundColor: annotation.style.backgroundColor,
    fontSize: `${annotation.style.fontSize}px`,
    fontFamily: annotation.style.fontFamily,
    fontWeight: annotation.style.fontWeight,
    fontStyle: annotation.style.fontStyle,
    textDecoration: annotation.style.textDecoration,
    textAlign: annotation.style.textAlign,
    lineHeight: "1.4",
    padding: "0.1em 0.2em",
    borderRadius: "4px",
  }}
>
  {annotation.content}
</span>

For figure rendering during export, annotationRenderer.ts invokes:

renderArrow(
  ctx,
  annotation.figureData.arrowDirection,
  annotation.figureData.color,
  annotation.figureData.strokeWidth,
  x,
  y,
  width,
  height,
  scaleFactor,
);

Summary

• AnnotationRegion objects in src/components/video-editor/types.ts provide a type-safe foundation for text, image, and figure overlays with full styling control through AnnotationTextStyle and FigureData.
• The three-layer rendering pipeline—canvas export (annotationRenderer.ts), live editing (AnnotationOverlay.tsx), and playback integration (VideoPlayback.tsx)—ensures consistent behavior across export and editing modes.
• Temporal control via startMs and endMs enables time-bound overlays that appear only during specific video segments.
• Z-index management allows precise stacking control, with automatic sorting by zIndex and selection boosting in the playback layer to handle overlapping regions.
• Figure overlays utilize pre-computed SVG path tables for directional arrows with customizable stroke width and color.

Frequently Asked Questions

How does OpenScreen handle overlapping annotation regions?

OpenScreen resolves overlapping annotations through z-index sorting. In both annotationRenderer.ts and VideoPlayback.tsx, regions are sorted by their zIndex property before rendering. During live editing, VideoPlayback.tsx implements click-through cycling that temporarily increases the selected annotation's z-index, ensuring users can select and edit regions even when they overlap completely.

What image formats are supported for image overlays?

OpenScreen accepts any image format that can be encoded as a data-URL (data:image/*). The content field of an image type AnnotationRegion expects a base64-encoded data string (e.g., data:image/png;base64,iVBORw0KG…). The renderer in annotationRenderer.ts uses standard Canvas API image loading, which supports PNG, JPEG, WebP, and other browser-compatible formats. The image is drawn using object-contain logic to preserve aspect ratio within the defined region bounds.

Can text annotations use custom fonts in the exported video?

Yes, provided the font is loaded in the browser context before export. The fontFamily property in AnnotationTextStyle accepts any CSS font family name. During canvas export in annotationRenderer.ts, the renderer applies the specified fontFamily to the canvas context. For consistent rendering between the live editor and final export, ensure custom fonts are preloaded using CSS @font-face or similar mechanisms before initiating the export process.

How are figure arrows positioned and oriented within their regions?

Figure arrows are positioned to fill the entire annotation region bounds. The figureData.arrowDirection property (e.g., "up-right", "down-left") selects a pre-computed SVG path from the ARROW_PATHS table in the renderer. The arrow is scaled to fit the region's width and height while maintaining stroke consistency according to strokeWidth. The color property defines the stroke color applied during canvas rendering in the renderArrow function.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}