# How OpenScreen Generates Zoom Suggestions from Cursor Movement

> Discover how OpenScreen analyzes cursor movement and dwell periods to automatically generate smart zoom suggestions during video playback for improved user engagement.

- Repository: [Sid/openscreen](https://github.com/siddharthvaddem/openscreen)
- Tags: internals
- Published: 2026-04-03

---

**OpenScreen analyzes cursor telemetry during video playback to detect dwell periods where the user pauses movement, then automatically proposes zoom regions centered on those points of interest.**

OpenScreen is an open-source video editor that implements intelligent **zoom suggestions** to streamline the editing workflow. The feature tracks mouse cursor movement while previewing footage and identifies moments where the user lingers as implicit signals for zoom targets. This data-driven approach eliminates manual guesswork when creating focus points in the timeline.

## Stage 1: Collecting and Normalizing Cursor Telemetry

In [`src/components/video-editor/timeline/zoomSuggestionUtils.ts`](https://github.com/siddharthvaddem/openscreen/blob/main/src/components/video-editor/timeline/zoomSuggestionUtils.ts), the `normalizeCursorTelemetry` function processes raw cursor data to ensure quality inputs for the suggestion engine.

### Removing Malformed Data

The function receives an array of `CursorTelemetryPoint` objects containing `timeMs`, `cx`, and `cy` coordinates. It removes malformed entries, clamps coordinates to a 0–1 normalized range, and sorts samples chronologically to prepare for sequential analysis.

## Stage 2: Detecting Dwell Candidates

The `detectZoomDwellCandidates` function in the same utility file scans the cleaned samples for pauses that indicate user interest.

### Movement Threshold Analysis

The algorithm uses `DWELL_MOVE_THRESHOLD = 0.02` to define the maximum cursor movement allowed within a dwell period. Consecutive samples staying within this threshold for longer than `MIN_DWELL_DURATION_MS` (450 ms) but shorter than `MAX_DWELL_DURATION_MS` (2600 ms) become `ZoomDwellCandidate` objects. Each candidate stores the centre time, average focus coordinates (`cx`, `cy`), and a **strength** value equal to the dwell duration in milliseconds.

## Stage 3: Ranking and Filtering Candidates

In [`src/components/video-editor/timeline/TimelineEditor.tsx`](https://github.com/siddharthvaddem/openscreen/blob/main/src/components/video-editor/timeline/TimelineEditor.tsx), the `handleSuggestZooms` function orchestrates the selection logic to prevent overlapping or redundant suggestions.

### Strength-Based Prioritization

Candidates sort by `strength` in descending order, prioritizing longer dwells that indicate higher user interest. The algorithm iterates through this ranked list and discards any candidate within `SUGGESTION_SPACING_MS` (1800 ms) of an already accepted centre to maintain temporal separation between zooms.

### Overlap Avoidance with Existing Zooms

Before finalizing a suggestion, the code checks against `reservedSpans`—the collection of existing manual zooms and previously accepted suggestions. It calculates a provisional zoom span (`defaultDuration/2` on each side of the centre time) and verifies no intersection with reserved regions. Only non-overlapping candidates proceed to emission.

## Stage 4: Emitting Zoom Suggestions

Approved candidates trigger the `onZoomSuggested(span, focus)` callback, passing the calculated time span and focus coordinates. The UI layer receives these parameters and displays a toast notification summarizing how many new zoom regions were added to the timeline.

## Implementation Example

The following TypeScript demonstrates the complete pipeline using OpenScreen's utility functions:

```typescript
import { 
  normalizeCursorTelemetry, 
  detectZoomDwellCandidates,
  CursorTelemetryPoint 
} from './zoomSuggestionUtils';

// Simulated telemetry collected during playback
const telemetry: CursorTelemetryPoint[] = [
  { timeMs: 1200, cx: 0.45, cy: 0.62 },
  { timeMs: 1300, cx: 0.46, cy: 0.61 },
  { timeMs: 1400, cx: 0.47, cy: 0.60 },
  // Additional samples...
];

// Step 1: Normalize input data
const normalized = normalizeCursorTelemetry(telemetry, videoDurationMs);

// Step 2: Identify dwell periods
const candidates = detectZoomDwellCandidates(normalized);

// Step 3: Select strongest candidates respecting constraints
const DEFAULT_DURATION = 2000; // Example duration in ms
const SUGGESTION_SPACING_MS = 1800;

const suggestions = candidates
  .sort((a, b) => b.strength - a.strength)
  .reduce<{ span: { start: number; end: number }; focus: { cx: number; cy: number } }[]>(
    (acc, candidate) => {
      // Check spacing from existing suggestions
      const tooClose = acc.some(s => 
        Math.abs(s.span.start + s.span.end - 2 * candidate.centerTimeMs) < SUGGESTION_SPACING_MS
      );
      if (tooClose) return acc;

      // Calculate span
      const start = Math.max(0, candidate.centerTimeMs - DEFAULT_DURATION / 2);
      const span = { start, end: start + DEFAULT_DURATION };
      
      acc.push({ span, focus: candidate.focus });
      return acc;
    }, 
    []
  );

// Step 4: Apply suggestions via callback
suggestions.forEach(s => onZoomSuggested(s.span, s.focus));

```

## Summary

- **Telemetry normalization** in [`zoomSuggestionUtils.ts`](https://github.com/siddharthvaddem/openscreen/blob/main/zoomSuggestionUtils.ts) sanitizes cursor coordinates and timestamps to ensure accurate analysis.
- **Dwell detection** uses a 0.02 movement threshold and 450–2600 ms duration window to identify implicit points of interest.
- **Smart filtering** ranks candidates by dwell length and enforces 1800 ms minimum spacing to prevent overcrowding.
- **Collision avoidance** checks against existing manual zooms to preserve user edits while adding automatic suggestions.

## Frequently Asked Questions

### How does OpenScreen determine where to place zoom suggestions?

OpenScreen tracks cursor telemetry during video playback and identifies **dwell periods** where the mouse remains within a 0.02 normalized distance threshold for 450–2600 ms. These pauses indicate user interest, and the system centers zoom suggestions on the average cursor position during these intervals.

### What prevents zoom suggestions from overlapping each other?

The algorithm in `handleSuggestZooms` enforces a `SUGGESTION_SPACING_MS` of 1800 ms between candidate centres. After sorting dwell candidates by strength (duration), it rejects any proposal that falls within 1800 ms of an already accepted suggestion, ensuring temporal separation in the timeline.

### Can zoom suggestions overlap with manually created zooms?

No. The system maintains a `reservedSpans` array tracking all existing zoom regions. Before emitting a suggestion, the code verifies that the proposed time span (centre time ± default duration) does not intersect with any reserved span, preserving manually placed zooms.

### What is the "strength" value in zoom dwell candidates?

The **strength** property equals the dwell duration in milliseconds. Longer pauses receive higher strength values, and the sorting algorithm prioritizes these candidates. This ensures that moments where the user lingers longest receive priority when generating zoom suggestions.