How OpenScreen Generates Zoom Suggestions from Cursor Movement

OpenScreen analyzes cursor telemetry during video playback to detect dwell periods where the user pauses movement, then automatically proposes zoom regions centered on those points of interest.

OpenScreen is an open-source video editor that implements intelligent zoom suggestions to streamline the editing workflow. The feature tracks mouse cursor movement while previewing footage and identifies moments where the user lingers as implicit signals for zoom targets. This data-driven approach eliminates manual guesswork when creating focus points in the timeline.

Stage 1: Collecting and Normalizing Cursor Telemetry

In src/components/video-editor/timeline/zoomSuggestionUtils.ts, the normalizeCursorTelemetry function processes raw cursor data to ensure quality inputs for the suggestion engine.

Removing Malformed Data

The function receives an array of CursorTelemetryPoint objects containing timeMs, cx, and cy coordinates. It removes malformed entries, clamps coordinates to a 0–1 normalized range, and sorts samples chronologically to prepare for sequential analysis.

Stage 2: Detecting Dwell Candidates

The detectZoomDwellCandidates function in the same utility file scans the cleaned samples for pauses that indicate user interest.

Movement Threshold Analysis

The algorithm uses DWELL_MOVE_THRESHOLD = 0.02 to define the maximum cursor movement allowed within a dwell period. Consecutive samples staying within this threshold for longer than MIN_DWELL_DURATION_MS (450 ms) but shorter than MAX_DWELL_DURATION_MS (2600 ms) become ZoomDwellCandidate objects. Each candidate stores the centre time, average focus coordinates (cx, cy), and a strength value equal to the dwell duration in milliseconds.

Stage 3: Ranking and Filtering Candidates

In src/components/video-editor/timeline/TimelineEditor.tsx, the handleSuggestZooms function orchestrates the selection logic to prevent overlapping or redundant suggestions.

Strength-Based Prioritization

Candidates sort by strength in descending order, prioritizing longer dwells that indicate higher user interest. The algorithm iterates through this ranked list and discards any candidate within SUGGESTION_SPACING_MS (1800 ms) of an already accepted centre to maintain temporal separation between zooms.

Overlap Avoidance with Existing Zooms

Before finalizing a suggestion, the code checks against reservedSpans—the collection of existing manual zooms and previously accepted suggestions. It calculates a provisional zoom span (defaultDuration/2 on each side of the centre time) and verifies no intersection with reserved regions. Only non-overlapping candidates proceed to emission.

Stage 4: Emitting Zoom Suggestions

Approved candidates trigger the onZoomSuggested(span, focus) callback, passing the calculated time span and focus coordinates. The UI layer receives these parameters and displays a toast notification summarizing how many new zoom regions were added to the timeline.

Implementation Example

The following TypeScript demonstrates the complete pipeline using OpenScreen's utility functions:

import { 
  normalizeCursorTelemetry, 
  detectZoomDwellCandidates,
  CursorTelemetryPoint 
} from './zoomSuggestionUtils';

// Simulated telemetry collected during playback
const telemetry: CursorTelemetryPoint[] = [
  { timeMs: 1200, cx: 0.45, cy: 0.62 },
  { timeMs: 1300, cx: 0.46, cy: 0.61 },
  { timeMs: 1400, cx: 0.47, cy: 0.60 },
  // Additional samples...
];

// Step 1: Normalize input data
const normalized = normalizeCursorTelemetry(telemetry, videoDurationMs);

// Step 2: Identify dwell periods
const candidates = detectZoomDwellCandidates(normalized);

// Step 3: Select strongest candidates respecting constraints
const DEFAULT_DURATION = 2000; // Example duration in ms
const SUGGESTION_SPACING_MS = 1800;

const suggestions = candidates
  .sort((a, b) => b.strength - a.strength)
  .reduce<{ span: { start: number; end: number }; focus: { cx: number; cy: number } }[]>(
    (acc, candidate) => {
      // Check spacing from existing suggestions
      const tooClose = acc.some(s => 
        Math.abs(s.span.start + s.span.end - 2 * candidate.centerTimeMs) < SUGGESTION_SPACING_MS
      );
      if (tooClose) return acc;

      // Calculate span
      const start = Math.max(0, candidate.centerTimeMs - DEFAULT_DURATION / 2);
      const span = { start, end: start + DEFAULT_DURATION };
      
      acc.push({ span, focus: candidate.focus });
      return acc;
    }, 
    []
  );

// Step 4: Apply suggestions via callback
suggestions.forEach(s => onZoomSuggested(s.span, s.focus));

Summary

• Telemetry normalization in zoomSuggestionUtils.ts sanitizes cursor coordinates and timestamps to ensure accurate analysis.
• Dwell detection uses a 0.02 movement threshold and 450–2600 ms duration window to identify implicit points of interest.
• Smart filtering ranks candidates by dwell length and enforces 1800 ms minimum spacing to prevent overcrowding.
• Collision avoidance checks against existing manual zooms to preserve user edits while adding automatic suggestions.

Frequently Asked Questions

How does OpenScreen determine where to place zoom suggestions?

OpenScreen tracks cursor telemetry during video playback and identifies dwell periods where the mouse remains within a 0.02 normalized distance threshold for 450–2600 ms. These pauses indicate user interest, and the system centers zoom suggestions on the average cursor position during these intervals.

What prevents zoom suggestions from overlapping each other?

The algorithm in handleSuggestZooms enforces a SUGGESTION_SPACING_MS of 1800 ms between candidate centres. After sorting dwell candidates by strength (duration), it rejects any proposal that falls within 1800 ms of an already accepted suggestion, ensuring temporal separation in the timeline.

Can zoom suggestions overlap with manually created zooms?

No. The system maintains a reservedSpans array tracking all existing zoom regions. Before emitting a suggestion, the code verifies that the proposed time span (centre time ± default duration) does not intersect with any reserved span, preserving manually placed zooms.

What is the "strength" value in zoom dwell candidates?

The strength property equals the dwell duration in milliseconds. Longer pauses receive higher strength values, and the sorting algorithm prioritizes these candidates. This ensures that moments where the user lingers longest receive priority when generating zoom suggestions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}