How to Integrate Three.js with Remotion: A Complete Guide

Remotion provides an official @remotion/three package that bridges React Three Fiber with Remotion's rendering pipeline through the <ThreeCanvas> component and useVideoTexture hook.

Integrating Three.js with Remotion enables you to render complex 3D scenes, animated meshes, and video textures directly into your programmatic videos. The remotion-dev/remotion repository ships a dedicated package that synchronizes React Three Fiber's render loop with Remotion's frame-based timeline, ensuring deterministic output across development and production renders.

Core Integration Primitives

The @remotion/three package exposes two essential building blocks that handle the bridge between Remotion's React environment and Three.js's WebGL renderer.

ThreeCanvas Component

<ThreeCanvas> wraps React Three Fiber's native <Canvas> component and enforces Remotion-specific constraints. Located in packages/three/src/ThreeCanvas.tsx, this component requires explicit width and height props to maintain deterministic sizing across server and client renders.

The component implements three critical behaviors:

• Forced dimensions: Renders a hidden Scale component (lines 25-38) that overrides the canvas size to match Remotion's video configuration
• Render loop control: Sets frameloop="never" (line 23) to disable automatic Three.js rendering
• Context propagation: Wraps children with Internals.RemotionContextProvider (lines 27-31) so Remotion hooks work inside the 3D scene

useVideoTexture Hook

useVideoTexture creates a Three.js VideoTexture that remains synchronized with Remotion's timeline. Implemented in packages/three/src/use-video-texture.ts, this hook accepts a React ref pointing to an HTML video element and returns a texture ready for Three.js materials.

The hook handles:

• Loading states: Calls delayRender (lines 34-41) to pause Remotion until the video metadata loads
• Texture creation: Instantiates new VideoTexture(videoRef.current) (line 66) with proper color space handling
• Frame synchronization: Registers requestVideoFrameCallback (lines 96-113) to advance Remotion's frame when the video updates

How the Integration Works

Understanding the internal mechanics ensures you can debug edge cases and optimize performance when integrating Three.js with Remotion.

Canvas Sizing and Scaling

Remotion requires deterministic dimensions for every frame to ensure server-side rendering matches the development preview. The <ThreeCanvas> component enforces this by accepting explicit width and height props rather than relying on CSS layout.

Internally, the Scale component (referenced in ThreeCanvas.tsx) forces the underlying React Three Fiber canvas to occupy exactly the specified pixel dimensions. This prevents a known browser bug where the canvas would otherwise report incorrect bounds during Remotion Studio playback, causing visual corruption in the final output.

Render Loop Control

Standard React Three Fiber applications run a continuous animation loop using requestAnimationFrame. This approach conflicts with Remotion's frame-by-frame rendering model, where each frame must be captured at a specific point in time.

The integration solves this by setting frameloop="never" on the underlying canvas. Instead of automatic updates, the ManualFrameRenderer component (lines 47-55 in the source) manually advances the Three.js scene using advance(performance.now()) whenever Remotion's useCurrentFrame() value changes. This guarantees that the WebGL buffer contains exactly the frame corresponding to the current timeline position when Remotion captures the screenshot.

Context Propagation

React Three Fiber relies on React context to share state between the canvas and 3D objects. Remotion also uses context to provide useCurrentFrame, useVideoConfig, and other hooks.

The <ThreeCanvas> component bridges these two context systems by rendering Internals.RemotionContextProvider as a child of the React Three Fiber scene. This allows any component inside the 3D tree—whether a <mesh>, <light>, or custom abstraction—to call Remotion hooks and react to timeline changes, enabling frame-perfect synchronization between 2D video elements and 3D geometry.

Video Texture Synchronization

Mapping video onto 3D surfaces requires careful handling of asynchronous video loading and frame timing. The useVideoTexture hook implements a robust synchronization mechanism.

When initialized, the hook immediately calls delayRender() to prevent Remotion from proceeding until the video element's metadata loads and the canplay event fires. Once ready, it creates a standard Three.js VideoTexture and configures it with the appropriate color space for accurate color reproduction.

The critical synchronization happens through requestVideoFrameCallback, a browser API that fires whenever the video decoder produces a new frame. The hook registers this callback (lines 96-113) and triggers a Remotion re-render each time the video advances, ensuring that the 3D texture updates in lock-step with the underlying video file and Remotion's timeline scrubbing.

Static Rendering Compatibility

Remotion supports server-side rendering in Node.js where no DOM or WebGL context exists. Both <ThreeCanvas> and useVideoTexture include runtime guards to handle this environment.

The components check for browser-only APIs before executing DOM-dependent code. For example, useVideoTexture verifies the existence of document and HTMLVideoElement before attempting to create textures or register video callbacks (lines 123-129). Similarly, <ThreeCanvas> ensures that Three.js initialization only occurs in browser contexts. This dual-environment support allows the same code to run during development (browser) and production rendering (Node.js/headless Chrome) without conditional imports or code splitting.

Practical Implementation Examples

These complete, runnable examples demonstrate common patterns for integrating Three.js with Remotion.

Rendering a Basic 3D Scene

This example creates a spinning cube with dynamic scaling and color changes, synchronized to Remotion's frame counter.

import {ThreeCanvas} from '@remotion/three';
import {interpolate, useCurrentFrame, useVideoConfig} from 'remotion';
import React from 'react';

export const ThreeBasic: React.FC = () => {
  const frame = useCurrentFrame();
  const {width, height} = useVideoConfig();

  return (
    <ThreeCanvas
      width={width}
      height={height}
      style={{backgroundColor: 'white'}}
      camera={{fov: 75, position: [0, 0, 470]}}
    >
      <ambientLight intensity={0.15} />
      <pointLight args={[undefined, 0.4]} position={[200, 200, 0]} />
      <mesh
        rotation={[
          frame * 0.06 * 0.5,
          frame * 0.07 * 0.5,
          frame * 0.08 * 0.5,
        ]}
        scale={interpolate(Math.sin(frame / 10), [-1, 1], [0.8, 1.2])}
      >
        <boxGeometry args={[100, 100, 100]} />
        <meshStandardMaterial
          color={[
            Math.sin(frame * 0.12) * 0.5 + 0.5,
            Math.cos(frame * 0.11) * 0.5 + 0.5,
            Math.sin(frame * 0.08) * 0.5 + 0.5,
          ]}
        />
      </mesh>
    </ThreeCanvas>
  );
};

This implementation references the official example in packages/example/src/ThreeBasic/index.tsx and demonstrates how useCurrentFrame drives animation inside the React Three Fiber scene.

Mapping Video Textures to 3D Objects

This example demonstrates the useVideoTexture hook to project video content onto a 3D cube surface.

import {ThreeCanvas, useVideoTexture} from '@remotion/three';
import {useCurrentFrame, useVideoConfig} from 'remotion';
import React, {useRef} from 'react';

export const VideoCube: React.FC = () => {
  const {width, height} = useVideoConfig();
  const videoRef = useRef<HTMLVideoElement>(null);
  const texture = useVideoTexture(videoRef);

  return (
    <ThreeCanvas width={width} height={height} camera={{position: [0, 0, 400]}}>
      <video
        ref={videoRef}
        src="my-video.mp4"
        style={{display: 'none'}}
        muted
        autoPlay
        loop
      />
      <mesh>
        <boxGeometry args={[150, 150, 150]} />
        {texture && <meshStandardMaterial map={texture} />}
      </mesh>
    </ThreeCanvas>
  );
};

The useVideoTexture hook automatically handles the delayRender logic (lines 34-41 in use-video-texture.ts) to pause Remotion until the video metadata loads, and registers requestVideoFrameCallback (lines 96-113) to synchronize texture updates with the video playback.

Combining 2D and 3D Elements

When mixing standard Remotion components with Three.js scenes, you must handle the DOM structure carefully to avoid invalid nesting.

import {Series, Sequence} from 'remotion';
import {ThreeCanvas} from '@remotion/three';
import {My3DScene} from './My3DScene';
import {MyLogo} from './MyLogo';

export const MixedComposition: React.FC = () => (
  <Series>
    <Sequence durationInFrames={150}>
      <MyLogo />
    </Sequence>

    <Sequence durationInFrames={300} layout="none">
      <ThreeCanvas width={1080} height={1080}>
        <My3DScene />
      </ThreeCanvas>
    </Sequence>
  </Series>
);

The layout="none" prop is required because <Sequence> normally renders a wrapping <div> element, which violates React Three Fiber's requirement that <ThreeCanvas> contain only Three.js-compatible children. This pattern is documented in packages/docs/docs/three-canvas.mdx (lines 57-60).

Key Source Files and Architecture

Understanding the implementation details in the remotion-dev/remotion repository helps debug advanced use cases and contributes to the ecosystem.

File	Purpose	Location
ThreeCanvas.tsx	Core wrapper component that synchronizes React Three Fiber with Remotion's frame loop.	packages/three/src/ThreeCanvas.tsx
use-video-texture.ts	Hook implementing video texture synchronization with delayRender and requestVideoFrameCallback.	packages/three/src/use-video-texture.ts
ThreeBasic/index.tsx	Minimal implementation example showing animated 3D geometry.	packages/example/src/ThreeBasic/index.tsx
ThreeScene/Scene.tsx	Advanced example demonstrating video textures in 3D space.	packages/example/src/ThreeScene/Scene.tsx
three-canvas.mdx	Official documentation and API reference.	packages/docs/docs/three-canvas.mdx

The architecture relies on manual frame advancement rather than continuous animation loops. In ThreeCanvas.tsx (line 23), the component sets frameloop="never" to disable automatic rendering, then uses advance(performance.now()) (lines 47-55) to step the Three.js simulation exactly when Remotion's useCurrentFrame() updates.

Summary

• Use @remotion/three to bridge React Three Fiber with Remotion's rendering pipeline instead of manually instantiating Three.js renderers.
• <ThreeCanvas> requires explicit width and height props and disables automatic frame loops to ensure deterministic, frame-perfect rendering synchronized with useCurrentFrame().
• useVideoTexture handles video-to-3D-texture mapping with automatic loading states and timeline synchronization via requestVideoFrameCallback.
• Server-side compatibility is built-in through runtime checks that guard against DOM-dependent code during Node.js rendering.
• Layout constraints require layout="none" on <Sequence> components when nesting <ThreeCanvas> to avoid invalid DOM nesting.

Frequently Asked Questions

What is the difference between using Three.js directly versus using @remotion/three?

Using Three.js directly requires manually creating a WebGL renderer, handling resize events, and synchronizing the animation loop with Remotion's frame-based timeline. The @remotion/three package handles these complexities automatically by wrapping React Three Fiber and enforcing frameloop="never" to ensure that Three.js only renders when Remotion advances the frame, guaranteeing deterministic output across development and production renders.

Why does my Three.js scene render black or fail to capture frames in Remotion Studio?

This typically occurs when the Three.js canvas attempts to auto-render outside of Remotion's control or when dimensions are not explicitly set. The <ThreeCanvas> component in packages/three/src/ThreeCanvas.tsx requires explicit width and height props and internally forces frameloop="never" (line 23). If you are using standard React Three Fiber <Canvas> without these safeguards, the render loop will desynchronize from Remotion's frame capture timing.

How do I synchronize a video file with a 3D texture in Remotion?

Use the useVideoTexture hook exported from @remotion/three. This hook accepts a React ref to a hidden <video> element and returns a Three.js VideoTexture. According to the implementation in packages/three/src/use-video-texture.ts, the hook automatically delays rendering until the video metadata loads (lines 34-41) and registers a requestVideoFrameCallback listener (lines 96-113) to ensure the texture updates in perfect synchronization with Remotion's timeline scrubbing.

Can I mix regular Remotion components with Three.js scenes in the same composition?

Yes, but you must handle the component hierarchy carefully. Standard Remotion components like <Sequence> render wrapper <div> elements by default, which invalidates the React Three Fiber tree if placed inside <ThreeCanvas>. When nesting <ThreeCanvas> inside a <Sequence>, always add the layout="none" prop to the <Sequence> component. This pattern, documented in packages/docs/docs/three-canvas.mdx (lines 57-60), removes the wrapping div and allows valid mixing of 2D and 3D elements using <Series> or nested <Sequence> components.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}