How Voicebox Stories Editor Timeline Composition Works: A Deep Dive

The Voicebox stories editor timeline composition converts millisecond-based audio durations into pixel coordinates using a configurable pixelsPerSecond scale, rendering clips on vertical tracks with synchronized playback via a Zustand store and React Query mutations.

The Voicebox repository (jamiepine/voicebox) implements a browser-based timeline editor for arranging voice-generated clips into structured stories. The stories editor timeline composition relies on a tightly-coupled React architecture that maps temporal audio data to spatial UI coordinates, enabling complex editing operations like trimming, splitting, and drag-and-drop rearrangement.

Core Architecture Components

The timeline implementation consists of four primary layers working in concert:

• StoryTrackEditor – The main UI component located in app/src/components/StoriesTab/StoryTrackEditor.tsx that orchestrates layout, zoom handling, mouse interactions, and playhead rendering.
• ClipWaveform – An inner component (lines 38-70) that instantiates Wavesurfer.js instances to render waveforms with trim offsets applied via CSS transforms.
• useStoryStore – A Zustand store defined in app/src/stores/storyStore.ts that manages reactive state for clip selection, editor height, and playback timing anchors.
• useStories hooks – React Query wrappers in app/src/lib/hooks/useStories.ts that expose mutations for CRUD operations including useMoveStoryItem, useTrimStoryItem, and useSplitStoryItem.

Timeline Geometry and Coordinate Mapping

Pixel-Per-Second Scaling

The editor establishes a bidirectional mapping between milliseconds and pixels through a configurable scale factor:

const [pixelsPerSecond, setPixelsPerSecond] = useState(DEFAULT_PIXELS_PER_SECOND);
// Where DEFAULT_PIXELS_PER_SECOND = 50 (line 30)

const handleZoomIn = () => setPixelsPerSecond(p => Math.min(p * 1.5, MAX_PIXELS_PER_SECOND));
const handleZoomOut = () => setPixelsPerSecond(p => Math.max(p / 1.5, MIN_PIXELS_PER_SECOND));

Zoom handlers multiply or divide the current scale by 1.5, clamping to defined minimum and maximum boundaries. This value drives all spatial calculations in the stories editor timeline composition.

Time-to-Pixel Conversion Functions

The component memoizes conversion helpers to translate between temporal and spatial units:

const msToPixels = useCallback((ms) => (ms / 1000) * pixelsPerSecond, [pixelsPerSecond]);
const pixelsToMs = useCallback((px) => (px / pixelsPerSecond) * 1000, [pixelsPerSecond]);

These functions calculate CSS left and width properties for clip positioning and convert mouse coordinates back to millisecond values during seek operations.

Dynamic Timeline Width Calculation

The total scrollable width derives from the longest clip endpoint plus padding:

const totalDurationMs = useMemo(() => {
  if (items.length === 0) return 10000; // default 10s
  return Math.max(...items.map(i => i.start_time_ms + getEffectiveDuration(i)), 10000);
}, [items, getEffectiveDuration]);

const contentWidth = (totalDurationMs / 1000) * pixelsPerSecond + 200; // extra padding
const timelineWidth = Math.max(contentWidth, containerWidth);

The getEffectiveDuration function accounts for trim offsets, ensuring the timeline extends beyond the last clip’s audible content.

Track Layout Management

Track Enumeration and Sorting

The editor maintains a vertical stack of tracks using a computed Set-based enumeration:

const tracks = useMemo(() => {
  const trackSet = new Set([...DEFAULT_TRACKS, ...items.map(i => i.track)]);
  return Array.from(trackSet).sort((a, b) => b - a); // higher index = top
}, [items]);

DEFAULT_TRACKS = [1, 0, -1] ensures three visible rows exist even in empty projects. Tracks render with a fixed TRACK_HEIGHT = 48 pixels (line 26).

Vertical Positioning

Each clip calculates its vertical offset based on track index:

const trackIndex = getTrackIndex(item.track);
const top = trackIndex * TRACK_HEIGHT;

The track index mapper converts track identifiers into zero-based indices for CSS top positioning, creating the layered visual layout characteristic of the Voicebox stories editor timeline composition.

Clip Rendering and Waveform Visualization

Effective Duration Calculations

Trim handles modify the audible portion of clips through duration arithmetic:

const getEffectiveDuration = (item) =>
  item.duration * 1000 - (item.trim_start_ms || 0) - (item.trim_end_ms || 0);

This value determines the rendered width via msToPixels, while the original duration determines the waveform container size.

Waveform Offset Handling

The ClipWaveform component renders full waveforms then offsets them to hide trimmed sections:

<div
  ref={waveformRef}
  style={{ 
    width: `${fullWaveformWidth}px`, 
    transform: `translateX(-${offsetX}px)` 
  }}
/>

Where offsetX equals the pixel value of trim_start_ms, effectively masking silent portions without regenerating the waveform.

User Interaction Handling

The timeline supports complex editing gestures through mouse event handlers:

• Drag-and-drop moving – handleDragStart (lines 47-62) captures mouse offsets relative to the clip; handleDragMove (lines 64-73) updates temporary dragPosition state for live visual feedback; handleDragEnd commits changes via the useMoveStoryItem mutation.
• Trimming – Trim handles appear on selected clips only. handleTrimMove updates tempTrimValues state for real-time width preview, while handleTrimEnd persists changes through useTrimStoryItem.
• Splitting – The S key or Split button triggers handleSplit (lines 502-525), calculating the split point relative to the playhead position and invoking useSplitStoryItem to create two clips from one.
• Keyboard shortcuts – Cmd/Ctrl+D duplicates via useDuplicateStoryItem, while Delete/Backspace removes clips through useRemoveStoryItem.

All mutations invalidate the stories React Query cache, ensuring UI synchronization with the server state.

Playback Synchronization and Auto-Scrolling

The store manages playback timing through anchored references:

// From storyStore.ts (line 56 onwards)
const play = () => {
  setPlaybackStartContextTime(audioContext.currentTime);
  setPlaybackStartStoryTime(currentTimeMs);
  setIsPlaying(true);
};

A useEffect hook (lines 47-61 in StoryTrackEditor) handles auto-scrolling during playback:

useEffect(() => {
  if (!isCurrentlyPlaying || !tracksRef.current) return;
  const container = tracksRef.current;
  const halfway = container.scrollLeft + container.clientWidth / 2;
  if (playheadLeft > halfway) {
    container.scrollLeft = playheadLeft - container.clientWidth / 2;
  }
}, [isCurrentlyPlaying, playheadLeft]);

This keeps the playhead centered in the viewport while audio plays, maintaining visual context during story review.

Implementation Example

Embed the editor in a parent view by providing the story ID and items array:

import { StoryTrackEditor } from '@/components/StoriesTab/StoryTrackEditor';
import { useStory } from '@/lib/hooks/useStories';

export function StoriesTab({ storyId }: { storyId: string }) {
  const { data: story } = useStory(storyId);
  
  if (!story) return null;
  
  return (
    <StoryTrackEditor
      storyId={story.id}
      items={story.items}
    />
  );
}

The component self-contains all geometry calculations, interaction handlers, and waveform generation, requiring only the story metadata and item array to render the full stories editor timeline composition.

Summary

• Coordinate system – Milliseconds convert to pixels via pixelsPerSecond scaling with configurable zoom bounds.
• Track layout – Vertical tracks render at 48-pixel heights, enumerated from a Set that merges default tracks with existing clip assignments.
• Waveform rendering – ClipWaveform uses Wavesurfer.js with CSS transforms to visually trim audio without processing.
• State management – Zustand handles UI state (selection, playback) while React Query manages server synchronization for mutations.
• Playback integration – Auto-scrolling centers the playhead during playback using timing anchors from the audio context.

Frequently Asked Questions

How does Voicebox convert between time and pixel coordinates?

The editor uses two memoized conversion functions, msToPixels and pixelsToMs, defined in StoryTrackEditor.tsx. These functions multiply or divide milliseconds by the current pixelsPerSecond value (default 50) divided by 1000, establishing a scalable coordinate system that updates when users zoom in or out.

What handles the drag-and-drop logic in the timeline?

Drag interactions follow a three-phase pattern: handleDragStart records initial mouse offsets and clip positions; handleDragMove updates temporary drag state for visual feedback; and handleDragEnd calculates the final millisecond destination and track index before calling the useMoveStoryItem mutation from useStories.ts.

How does the timeline handle audio trimming visually?

Trimming manipulates the transform: translateX CSS property on the waveform container. The ClipWaveform component renders the full waveform at original duration width, then offsets it by the trim_start_ms value converted to pixels, effectively hiding trimmed sections without regenerating the waveform data.

Where is the playback state managed in Voicebox?

Playback state resides in the Zustand store at app/src/stores/storyStore.ts, which maintains currentTimeMs, isPlaying, and timing anchors. The StoryTrackEditor reads these values to position the playhead and trigger auto-scrolling, while audio synchronization occurs through a separate hook that updates the store on each animation frame.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}