What Is Cursor Telemetry in OpenScreen? Mouse Movement Tracking and Replay Explained

Cursor telemetry in OpenScreen is a time-series of normalized mouse coordinates captured every 100ms during screen recording, stored as JSON alongside video files, and consumed by the React frontend to replay cursor positions or generate automatic zoom suggestions based on dwell detection.

OpenScreen, the open-source screen recording tool maintained at siddharthvaddem/openscreen, implements cursor telemetry to track mouse movements during capture sessions. This system samples cursor positions at regular intervals, persists them as resolution-independent coordinates, and enables both visualization of cursor history and intelligent editing features. The telemetry pipeline bridges Electron's main process APIs with the React renderer to provide seamless mouse tracking across different display configurations.

How Cursor Telemetry Is Captured in OpenScreen

The capture pipeline begins when a recording session starts via the set-recording-state IPC handler in electron/ipc/handlers.ts. This handler initializes the telemetry collection by clearing previous samples, recording a start timestamp, and scheduling a periodic timer using CURSOR_SAMPLE_INTERVAL_MS = 100.

Every 100 milliseconds, the sampleCursorPoint() function executes the following sequence:

1. Reads the absolute cursor position using Electron's screen.getCursorScreenPoint()
2. Identifies the active display containing that point
3. Converts absolute screen pixels to normalized coordinates (cx, cy) where values range from 0 to 1 (representing the percentage of screen width and height)
4. Appends a CursorTelemetryPoint object containing timeMs (milliseconds since recording start), cx, and cy to the activeCursorSamples array

When recording stops, the timer clears and the accumulated samples move to pendingCursorSamples. The storeRecordedSessionFiles() function then persists this array to a JSON file adjacent to the video recording at ${screenVideoPath}.cursor.json.

The Cursor Telemetry Data Structure

OpenScreen stores cursor data using the CursorTelemetryPoint interface to ensure type safety and consistent normalization:

export interface CursorTelemetryPoint {
  timeMs: number; // milliseconds since recording start
  cx: number;     // normalized X (0 = left, 1 = right)
  cy: number;     // normalized Y (0 = top, 1 = bottom)
}

The persisted JSON file follows this structure:

{
  "version": 1,
  "samples": [
    { "timeMs": 0,   "cx": 0.45, "cy": 0.62 },
    { "timeMs": 100, "cx": 0.46, "cy": 0.63 },
    { "timeMs": 200, "cx": 0.47, "cy": 0.63 }
  ]
}

Normalized coordinates ensure that cursor telemetry remains accurate regardless of the display resolution or DPI settings used during recording. By storing positions as proportions rather than absolute pixels, the same telemetry file can replay correctly on monitors with different resolutions—the UI simply multiplies cx by the target video width and cy by the video height at render time.

Replaying Mouse Movements from Telemetry

To consume cursor telemetry in the React frontend, the application uses the get-cursor-telemetry IPC channel exposed through the Electron preload script. The main process handler in electron/ipc/handlers.ts (lines 91-130) loads the JSON file, validates the data, and returns normalized samples to the renderer.

The VideoEditor component in src/components/video-editor/VideoEditor.tsx fetches telemetry using this pattern:

import { useEffect, useState } from "react";
import type { CursorTelemetryPoint } from "../types";

function useCursorTelemetry(videoPath: string | null) {
  const [telemetry, setTelemetry] = useState<CursorTelemetryPoint[]>([]);

  useEffect(() => {
    if (!videoPath) {
      setTelemetry([]);
      return;
    }

    window.electronAPI
      .getCursorTelemetry(videoPath)
      .then((result) => {
        if (result.success) setTelemetry(result.samples);
      })
      .catch((e) => console.warn("Unable to load cursor telemetry:", e));
  }, [videoPath]);

  return telemetry;
}

You can leverage this data to render a cursor overlay during video playback. This React component demonstrates how to position an element based on the current playback time:

import { useRef, useEffect } from "react";

function CursorOverlay({
  telemetry,
  currentTimeSec,
  videoWidth,
  videoHeight,
}: {
  telemetry: CursorTelemetryPoint[];
  currentTimeSec: number;
  videoWidth: number;
  videoHeight: number;
}) {
  const overlayRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    if (!overlayRef.current) return;

    const target = telemetry.reduce((best, p) => {
      const dt = Math.abs(p.timeMs / 1000 - currentTimeSec);
      return dt < best.dt ? { point: p, dt } : best;
    }, { point: null as CursorTelemetryPoint | null, dt: Infinity });

    if (!target.point) {
      overlayRef.current.style.display = "none";
      return;
    }

    const { cx, cy } = target.point;
    overlayRef.current.style.left = `${cx * videoWidth}px`;
    overlayRef.current.style.top = `${cy * videoHeight}px`;
    overlayRef.current.style.display = "block";
  }, [currentTimeSec, telemetry, videoWidth, videoHeight]);

  return (
    <div
      ref={overlayRef}
      style={{
        position: "absolute",
        width: "16px",
        height: "16px",
        background: "rgba(255,255,0,0.8)",
        borderRadius: "50%",
        pointerEvents: "none",
        transform: "translate(-50%,-50%)",
      }}
    />
  );
}

Smart Zoom Suggestions Using Dwell Detection

OpenScreen uses cursor telemetry to automatically suggest zoom regions where users linger during recordings. This dwell detection algorithm resides in src/components/video-editor/timeline/zoomSuggestionUtils.ts.

The TimelineEditor component processes telemetry through two key utilities:

1. Normalization: normalizeCursorTelemetry() clamps coordinate values between 0 and 1, ensures finite numbers, and sorts samples chronologically
2. Dwell Analysis: detectZoomDwellCandidates() groups consecutive points that remain within a DWELL_MOVE_THRESHOLD of 0.02 (2% of screen dimensions) for durations between 450ms and 2600ms

Each detected dwell generates a ZoomDwellCandidate containing the center timestamp, averaged focus coordinates, and a strength value proportional to the dwell duration. The UI presents these as suggested zoom markers on the timeline.

Implementation follows this pattern:

import {
  normalizeCursorTelemetry,
  detectZoomDwellCandidates,
} from "./zoomSuggestionUtils";

function suggestZooms(
  telemetry: CursorTelemetryPoint[],
  videoDurationSec: number,
) {
  const totalMs = Math.round(videoDurationSec * 1000);
  const normalized = normalizeCursorTelemetry(telemetry, totalMs);
  return detectZoomDwellCandidates(normalized);
}

Summary

• Cursor telemetry in OpenScreen consists of time-stamped, normalized mouse coordinates captured every 100ms during screen recording sessions.
• The sampling logic in electron/ipc/handlers.ts converts absolute screen pixels to relative coordinates (0-1) using sampleCursorPoint(), enabling resolution-independent replay.
• Telemetry persists as JSON files alongside video recordings (${videoPath}.cursor.json) and loads via the get-cursor-telemetry IPC handler.
• The React frontend consumes this data through VideoEditor.tsx and TimelineEditor.tsx to support cursor visualization and automated zoom suggestions.
• Dwell detection analyzes telemetry clusters to identify where users pause, generating smart zoom candidates using thresholds defined in zoomSuggestionUtils.ts.

Frequently Asked Questions

How often does OpenScreen sample cursor positions?

OpenScreen samples cursor positions every 100 milliseconds (10Hz) using a timer scheduled when recording begins. This frequency balances data granularity with storage efficiency, capturing sufficient movement detail while keeping JSON file sizes manageable for long recordings.

Can cursor telemetry be used across different screen resolutions?

Yes. OpenScreen stores coordinates as normalized values (0.0 to 1.0) representing the percentage of screen width and height. When replaying or rendering overlays, the UI multiplies these factors by the current video dimensions, ensuring accurate cursor positioning regardless of the original or display resolution.

What is the file format for stored cursor telemetry?

Cursor telemetry stores as a JSON file with a .cursor.json extension appended to the video file path. The file contains a version field (currently 1) and a samples array with objects specifying timeMs, cx, and cy properties. This format enables easy inspection, debugging, and potential third-party consumption.

How does OpenScreen detect zoom suggestions from cursor data?

The application detects zoom suggestions by analyzing dwell patterns in the telemetry data. Consecutive samples remaining within a 2% screen radius (DWELL_MOVE_THRESHOLD = 0.02) for 450ms to 2600ms trigger candidate generation. The algorithm averages these clustered coordinates to suggest focus points, helping editors automatically identify regions where users likely intended emphasis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}