# What Is Cursor Telemetry in OpenScreen? Mouse Movement Tracking and Replay Explained

> Discover cursor telemetry in OpenScreen. Learn how this mouse movement tracking and replay feature captures and stores coordinate data for enhanced screen recording analysis.

- Repository: [Sid/openscreen](https://github.com/siddharthvaddem/openscreen)
- Tags: deep-dive
- Published: 2026-04-03

---

**Cursor telemetry in OpenScreen is a time-series of normalized mouse coordinates captured every 100ms during screen recording, stored as JSON alongside video files, and consumed by the React frontend to replay cursor positions or generate automatic zoom suggestions based on dwell detection.**

OpenScreen, the open-source screen recording tool maintained at `siddharthvaddem/openscreen`, implements **cursor telemetry** to track mouse movements during capture sessions. This system samples cursor positions at regular intervals, persists them as resolution-independent coordinates, and enables both visualization of cursor history and intelligent editing features. The telemetry pipeline bridges Electron's main process APIs with the React renderer to provide seamless mouse tracking across different display configurations.

## How Cursor Telemetry Is Captured in OpenScreen

The capture pipeline begins when a recording session starts via the `set-recording-state` IPC handler in [`electron/ipc/handlers.ts`](https://github.com/siddharthvaddem/openscreen/blob/main/electron/ipc/handlers.ts). This handler initializes the telemetry collection by clearing previous samples, recording a start timestamp, and scheduling a periodic timer using `CURSOR_SAMPLE_INTERVAL_MS = 100`.

Every 100 milliseconds, the `sampleCursorPoint()` function executes the following sequence:

1. Reads the absolute cursor position using Electron's `screen.getCursorScreenPoint()`
2. Identifies the active display containing that point
3. Converts absolute screen pixels to **normalized coordinates** (`cx`, `cy`) where values range from 0 to 1 (representing the percentage of screen width and height)
4. Appends a `CursorTelemetryPoint` object containing `timeMs` (milliseconds since recording start), `cx`, and `cy` to the `activeCursorSamples` array

When recording stops, the timer clears and the accumulated samples move to `pendingCursorSamples`. The `storeRecordedSessionFiles()` function then persists this array to a JSON file adjacent to the video recording at `${screenVideoPath}.cursor.json`.

## The Cursor Telemetry Data Structure

OpenScreen stores cursor data using the `CursorTelemetryPoint` interface to ensure type safety and consistent normalization:

```typescript
export interface CursorTelemetryPoint {
  timeMs: number; // milliseconds since recording start
  cx: number;     // normalized X (0 = left, 1 = right)
  cy: number;     // normalized Y (0 = top, 1 = bottom)
}

```

The persisted JSON file follows this structure:

```json
{
  "version": 1,
  "samples": [
    { "timeMs": 0,   "cx": 0.45, "cy": 0.62 },
    { "timeMs": 100, "cx": 0.46, "cy": 0.63 },
    { "timeMs": 200, "cx": 0.47, "cy": 0.63 }
  ]
}

```

**Normalized coordinates** ensure that cursor telemetry remains accurate regardless of the display resolution or DPI settings used during recording. By storing positions as proportions rather than absolute pixels, the same telemetry file can replay correctly on monitors with different resolutions—the UI simply multiplies `cx` by the target video width and `cy` by the video height at render time.

## Replaying Mouse Movements from Telemetry

To consume cursor telemetry in the React frontend, the application uses the `get-cursor-telemetry` IPC channel exposed through the Electron preload script. The main process handler in [`electron/ipc/handlers.ts`](https://github.com/siddharthvaddem/openscreen/blob/main/electron/ipc/handlers.ts) (lines 91-130) loads the JSON file, validates the data, and returns normalized samples to the renderer.

The `VideoEditor` component in [`src/components/video-editor/VideoEditor.tsx`](https://github.com/siddharthvaddem/openscreen/blob/main/src/components/video-editor/VideoEditor.tsx) fetches telemetry using this pattern:

```typescript
import { useEffect, useState } from "react";
import type { CursorTelemetryPoint } from "../types";

function useCursorTelemetry(videoPath: string | null) {
  const [telemetry, setTelemetry] = useState<CursorTelemetryPoint[]>([]);

  useEffect(() => {
    if (!videoPath) {
      setTelemetry([]);
      return;
    }

    window.electronAPI
      .getCursorTelemetry(videoPath)
      .then((result) => {
        if (result.success) setTelemetry(result.samples);
      })
      .catch((e) => console.warn("Unable to load cursor telemetry:", e));
  }, [videoPath]);

  return telemetry;
}

```

You can leverage this data to render a cursor overlay during video playback. This React component demonstrates how to position an element based on the current playback time:

```tsx
import { useRef, useEffect } from "react";

function CursorOverlay({
  telemetry,
  currentTimeSec,
  videoWidth,
  videoHeight,
}: {
  telemetry: CursorTelemetryPoint[];
  currentTimeSec: number;
  videoWidth: number;
  videoHeight: number;
}) {
  const overlayRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    if (!overlayRef.current) return;

    const target = telemetry.reduce((best, p) => {
      const dt = Math.abs(p.timeMs / 1000 - currentTimeSec);
      return dt < best.dt ? { point: p, dt } : best;
    }, { point: null as CursorTelemetryPoint | null, dt: Infinity });

    if (!target.point) {
      overlayRef.current.style.display = "none";
      return;
    }

    const { cx, cy } = target.point;
    overlayRef.current.style.left = `${cx * videoWidth}px`;
    overlayRef.current.style.top = `${cy * videoHeight}px`;
    overlayRef.current.style.display = "block";
  }, [currentTimeSec, telemetry, videoWidth, videoHeight]);

  return (
    <div
      ref={overlayRef}
      style={{
        position: "absolute",
        width: "16px",
        height: "16px",
        background: "rgba(255,255,0,0.8)",
        borderRadius: "50%",
        pointerEvents: "none",
        transform: "translate(-50%,-50%)",
      }}
    />
  );
}

```

## Smart Zoom Suggestions Using Dwell Detection

OpenScreen uses cursor telemetry to automatically suggest zoom regions where users linger during recordings. This **dwell detection** algorithm resides in [`src/components/video-editor/timeline/zoomSuggestionUtils.ts`](https://github.com/siddharthvaddem/openscreen/blob/main/src/components/video-editor/timeline/zoomSuggestionUtils.ts).

The `TimelineEditor` component processes telemetry through two key utilities:

1. **Normalization**: `normalizeCursorTelemetry()` clamps coordinate values between 0 and 1, ensures finite numbers, and sorts samples chronologically
2. **Dwell Analysis**: `detectZoomDwellCandidates()` groups consecutive points that remain within a `DWELL_MOVE_THRESHOLD` of 0.02 (2% of screen dimensions) for durations between **450ms** and **2600ms**

Each detected dwell generates a `ZoomDwellCandidate` containing the center timestamp, averaged focus coordinates, and a strength value proportional to the dwell duration. The UI presents these as suggested zoom markers on the timeline.

Implementation follows this pattern:

```typescript
import {
  normalizeCursorTelemetry,
  detectZoomDwellCandidates,
} from "./zoomSuggestionUtils";

function suggestZooms(
  telemetry: CursorTelemetryPoint[],
  videoDurationSec: number,
) {
  const totalMs = Math.round(videoDurationSec * 1000);
  const normalized = normalizeCursorTelemetry(telemetry, totalMs);
  return detectZoomDwellCandidates(normalized);
}

```

## Summary

- **Cursor telemetry** in OpenScreen consists of time-stamped, normalized mouse coordinates captured every 100ms during screen recording sessions.
- The sampling logic in [`electron/ipc/handlers.ts`](https://github.com/siddharthvaddem/openscreen/blob/main/electron/ipc/handlers.ts) converts absolute screen pixels to relative coordinates (0-1) using `sampleCursorPoint()`, enabling resolution-independent replay.
- Telemetry persists as JSON files alongside video recordings (`${videoPath}.cursor.json`) and loads via the `get-cursor-telemetry` IPC handler.
- The React frontend consumes this data through [`VideoEditor.tsx`](https://github.com/siddharthvaddem/openscreen/blob/main/VideoEditor.tsx) and [`TimelineEditor.tsx`](https://github.com/siddharthvaddem/openscreen/blob/main/TimelineEditor.tsx) to support cursor visualization and automated zoom suggestions.
- **Dwell detection** analyzes telemetry clusters to identify where users pause, generating smart zoom candidates using thresholds defined in [`zoomSuggestionUtils.ts`](https://github.com/siddharthvaddem/openscreen/blob/main/zoomSuggestionUtils.ts).

## Frequently Asked Questions

### How often does OpenScreen sample cursor positions?

OpenScreen samples cursor positions every **100 milliseconds** (10Hz) using a timer scheduled when recording begins. This frequency balances data granularity with storage efficiency, capturing sufficient movement detail while keeping JSON file sizes manageable for long recordings.

### Can cursor telemetry be used across different screen resolutions?

Yes. OpenScreen stores coordinates as **normalized values** (0.0 to 1.0) representing the percentage of screen width and height. When replaying or rendering overlays, the UI multiplies these factors by the current video dimensions, ensuring accurate cursor positioning regardless of the original or display resolution.

### What is the file format for stored cursor telemetry?

Cursor telemetry stores as a JSON file with a [`.cursor.json`](https://github.com/siddharthvaddem/openscreen/blob/main/.cursor.json) extension appended to the video file path. The file contains a version field (currently `1`) and a samples array with objects specifying `timeMs`, `cx`, and `cy` properties. This format enables easy inspection, debugging, and potential third-party consumption.

### How does OpenScreen detect zoom suggestions from cursor data?

The application detects zoom suggestions by analyzing **dwell patterns** in the telemetry data. Consecutive samples remaining within a 2% screen radius (`DWELL_MOVE_THRESHOLD = 0.02`) for 450ms to 2600ms trigger candidate generation. The algorithm averages these clustered coordinates to suggest focus points, helping editors automatically identify regions where users likely intended emphasis.