How Deep-Interactive Mode Renders 3D Visualizations and Simulations in OpenMAIC

Deep-interactive mode renders 3D visualizations and simulations by attaching a widget to the page that uses a lightweight Three.js engine loaded on-demand from a CDN, converting JSON scene outlines into interactive client-side WebGL scenes.

The OpenMAIC platform (THU-MAIC/OpenMAIC) enables immersive educational content through its deep-interactive mode, which transforms static course materials into hands-on 3D experiences. This system relies on a clear separation between server-side scene generation and client-side rendering, ensuring that complex 3D molecular models, solar system simulations, or anatomical visualizations load efficiently while remaining fully interactive for learners.

Widget Definition and Scene Outlines

Every interactive page in deep-interactive mode requires a widget outline defined in JSON format. According to the SKILL.md file located at skills/agent-runtime/deep-interactive/SKILL.md, authors specify the widget type and configuration using a widgetOutline object that describes the entire 3D scene structure.

For 3D visualizations, the outline includes:

• visualizationType: A categorical hint such as molecular, solar, or anatomy that guides the renderer
• objects: An array of geometric descriptors containing geometry types, materials, and spatial positions
• interactions: UI control definitions including sliders, buttons, and camera manipulation options

The TypeScript definitions in lib/types/widgets.ts enforce type safety for these structures, distinguishing between visualization3d widgets (for static inspection) and simulation widgets (for parameter-driven experiments).

{
  "widgetType": "visualization3d",
  "widgetOutline": {
    "visualizationType": "molecular",
    "objects": [
      { "type": "sphere", "radius": 1, "material": "phong", "position": [0,0,0] },
      { "type": "cylinder", "radiusTop": 0.2, "radiusBottom": 0.2,
        "height": 3, "material": "lambert", "position": [2,0,0] }
    ],
    "interactions": [
      { "control": "rotate", "axis": "y" },
      { "control": "zoom", "min": 0.5, "max": 5 }
    ]
  }
}

Server-Side Scene Generation

The transformation from outline to renderable scene occurs in packages/@openmaic/generation/src/scene-generator.ts. This scene generator receives the widgetOutline JSON and constructs a compact payload describing the complete Three.js scene graph, including initial camera positions, lighting configurations, and interactive control bindings.

The generated payload flows through the scene-outlines-stream API endpoint defined in app/api/generate/scene-outlines-stream/route.ts, which streams the scene description to the client workbench. Authors edit these outlines using the React component found in components/generation/outlines-editor.tsx, which provides a visual interface for adjusting object properties and interaction parameters without manually writing JSON.

Client-Side Rendering with Three.js

When a learner opens an interactive page, the workbench dynamically imports the Three.js library from https://unpkg.com/three@…/build/three.module.js. The rendering component—exemplified by implementations such as components/scene-renderers/pbl/v2/sidebar.tsx—parses the JSON payload and constructs the 3D environment programmatically.

The client-side initialization follows this sequence:

1. Create a Three.js Scene object and configure WebGL renderer with antialiasing
2. Instantiate PerspectiveCamera with parameters from the server payload
3. Iterate through the objects array to generate Mesh instances with appropriate geometries (Sphere, Cylinder, etc.) and materials (Phong, Lambert)
4. Attach OrbitControls to enable camera rotation, zoom, and panning for visualization3d widgets
5. Register the animation loop using requestAnimationFrame for continuous rendering

import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';

export default function Visualization3D({ payload }: { payload: any }) {
  const mountRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    const scene = new THREE.Scene();
    const camera = new THREE.PerspectiveCamera(75, 1, 0.1, 1000);
    const renderer = new THREE.WebGLRenderer({ antialias: true });
    renderer.setSize(400, 400);
    mountRef.current?.appendChild(renderer.domElement);

    // add objects from payload
    payload.objects.forEach((obj: any) => {
      let mesh;
      if (obj.type === 'sphere') {
        const geo = new THREE.SphereGeometry(obj.radius, 32, 32);
        const mat = new THREE.MeshPhongMaterial({ color: 0x156289 });
        mesh = new THREE.Mesh(geo, mat);
      } else if (obj.type === 'cylinder') {
        const geo = new THREE.CylinderGeometry(
          obj.radiusTop,
          obj.radiusBottom,
          obj.height,
          32
        );
        const mat = new THREE.MeshLambertMaterial({ color: 0x8ac });
        mesh = new THREE.Mesh(geo, mat);
      }
      mesh.position.set(...obj.position);
      scene.add(mesh);
    });

    const controls = new OrbitControls(camera, renderer.domElement);
    camera.position.set(5, 5, 5);
    controls.update();

    const animate = () => {
      requestAnimationFrame(animate);
      renderer.render(scene, camera);
    };
    animate();
  }, [payload]);

  return <div ref={mountRef} />;
}

Simulation Interactions and Real-Time Updates

For simulation widgets, the generated payload includes keyVariables that map directly to UI controls such as sliders and toggles. When a learner modifies a variable, the React component updates the Three.js scene in real time by manipulating object properties within the animation loop.

The update mechanism typically references objects by name and applies transformations based on the current variable values:

function updateSimulation(variableName: string, value: number) {
  // payload contains a reference to the mesh or uniform; we update it here
  const target = scene.getObjectByName(variableName);
  if (target) {
    target.scale.setScalar(value);
  }
}

This approach allows parameter-driven experiments where changing a slider immediately affects the 3D visualization, creating a responsive learning environment for exploring scientific models.

Fallback Handling for Offline Browsers

The deep-interactive mode implements a safety fallback for scenarios where the browser cannot load Three.js, such as offline environments or restrictive network conditions. When the CDN request fails, the workbench automatically displays a static placeholder image accompanied by explanatory text, ensuring learners always receive educational content rather than encountering broken interactive elements.

Summary

• Widget outlines defined in SKILL.md structure specify 3D scenes using JSON descriptors for geometry, materials, and interactions.
• The scene generator (scene-generator.ts) processes these outlines server-side to create optimized JSON payloads for the client.
• Three.js loads on-demand from a CDN and executes within React components to render WebGL scenes with camera controls and lighting.
• Simulation widgets support real-time parameter updates through keyVariables that bind UI controls to Three.js object properties.
• Graceful degradation ensures content remains accessible via static fallbacks when WebGL or network resources are unavailable.

Frequently Asked Questions

How does OpenMAIC handle performance for complex 3D scenes?

The platform optimizes performance by generating compact JSON payloads server-side that contain only essential Three.js configuration data, minimizing transfer overhead. The client-side renderer uses standard optimization techniques such as geometry instancing and antialiased WebGL canvases, while offloading heavy computation to the Three.js animation loop running at the browser's native refresh rate.

What happens if a learner's browser does not support WebGL?

If the browser fails to load Three.js from the CDN or lacks WebGL support, the deep-interactive mode automatically falls back to displaying a static placeholder image with explanatory text. This ensures educational continuity without exposing users to broken interactive widgets or error states.

Can authors customize the interaction controls for specific visualization types?

Yes, authors define interaction behaviors within the interactions array of the widgetOutline JSON structure. They can specify controls for camera rotation, zoom constraints, and simulation variables that the outlines-editor.tsx component renders as intuitive UI widgets, allowing precise tailoring of the learning experience to specific domains like molecular biology or astronomy.

Is the Three.js library bundled with OpenMAIC or loaded externally?

The Three.js engine loads externally from a CDN (unpkg.com) on-demand when a user opens an interactive page. This approach reduces the initial bundle size of the OpenMAIC workbench while ensuring users always receive the specified version of the 3D library configured for their scene.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}