How Spatial Computing Agents Handle Immersive Interface Development: A Three-Layer Pipeline
Spatial computing agents in the msitarzewski/agency-agents repository employ a modular, role-driven pipeline where specialized agents handle UX architecture, WebXR prototyping, and native Metal or VisionOS rendering to deliver cross-platform immersive experiences.
The spatial computing agents form a comprehensive architecture that covers everything from ergonomic 3D interface design to high-performance native rendering. By combining specialized expertise across interaction design, application logic, and platform integration, these agents enable the development of immersive interfaces that run in browsers, on VisionOS, or as Metal-based renderers on macOS and Apple Vision Pro.
The Three-Layer Architecture of Spatial Computing Agents
The repository organizes spatial computing agents into three distinct layers, each owning a specific slice of the immersive-interface stack.
Interaction and UX Layer
The XR Interface Architect and XR Cockpit Interaction Specialist handle spatial layout and ergonomic design. According to spatial-computing/xr-interface-architect.md, the architect drafts comfortable 3D UI flows, defines interaction zones, and produces layout templates that mitigate motion sickness. The cockpit specialist, documented in spatial-computing/xr-cockpit-interaction-specialist.md, extends these designs with fixed-perspective controls such as yokes, levers, and gauges, encoding ergonomic rules for multimodal input patterns including gaze, pinch, and hand-tracking.
Application Logic and Rendering Layer
The XR Immersive Developer and macOS Spatial/Metal Engineer manage core scene graphs and performance-critical rendering. As implemented in spatial-computing/xr-immersive-developer.md, the immersive developer scaffolds cross-platform WebXR projects using Three.js or Babylon.js, wiring ray-casting, hit-testing, and physics validation. For workloads requiring sustained performance above 90 fps on massive datasets, the macOS Spatial/Metal Engineer—defined in spatial-computing/macos-spatial-metal-engineer.md—replaces WebGL renderers with native Metal pipelines, streaming frames to Vision Pro via RemoteImmersiveSpace.
Platform Integration Layer
The visionOS Spatial Engineer handles native Apple ecosystem integration. According to spatial-computing/visionos-spatial-engineer.md, this agent wraps scenes in SwiftUI WindowGroup containers, applies Apple's Liquid-Glass design system through modifiers like glassBackgroundEffect(.ultraThin), and implements accessibility hooks for VoiceOver and Switch Control. The engineer also manages CompositorServices for remote-immersive streaming when paired with Metal backends.
Architectural Flow: From Concept to Deployment
The spatial computing agents follow a seven-stage pipeline that transforms experience concepts into deployable immersive applications.
-
Concept and UX Definition – The XR Interface Architect drafts UI flows and produces layout templates stored as reusable components.
-
Interaction Specification – The XR Cockpit Interaction Specialist extends designs with cockpit-style controls and encodes ergonomic constraints for spatial input.
-
Web-Based Prototyping – The XR Immersive Developer scaffolds a WebXR project in
src/main.js, implementingrequestHitTestSourcefor ray-casting and validating the experience across Meta Quest, iOS browsers, and desktop. -
High-Performance Native Build – For demanding workloads, the macOS Spatial/Metal Engineer implements a
MetalGraphRendererclass with instanced rendering, replacing the WebGL pipeline to achieve 90+ fps on large node graphs. -
VisionOS Integration – The visionOS Spatial Engineer wraps the scene in a
RealityView, appliesglassBackgroundEffect(.ultraThin)for Liquid-Glass styling, and managesWindowGrouppresentation. -
Orchestration – The Agents Orchestrator (
specialized/agents-orchestrator.md) coordinates hand-offs, passing artifacts such as UI specs, component libraries, and shaders between agents while tracking dependencies. -
Deployment and Monitoring – Final bundles deploy as WebXR sites, VisionOS apps, or macOS Metal applications, tracked through strategy runbooks (
strategy/playbooks/phase-3-build.md).
Implementation Examples
WebXR Prototyping with Three.js
The XR Immersive Developer creates entry points using the WebXR Device API with Three.js. The onSelect handler demonstrates hit-testing implementation:
// src/main.js – minimal WebXR entry point
import { XRButton } from 'three/examples/jsm/webxr/XRButton.js';
import { Scene, PerspectiveCamera, WebGLRenderer } from 'three';
const scene = new Scene();
const camera = new PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 100);
const renderer = new WebGLRenderer({ antialias: true });
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.xr.enabled = true;
document.body.appendChild(renderer.domElement);
document.body.appendChild(XRButton.createButton(renderer));
// Simple hit-test raycast
function onSelect(event) {
const referenceSpace = renderer.xr.getReferenceSpace();
const session = renderer.xr.getSession();
session.requestHitTestSource({ space: referenceSpace }).then(src => {
const hit = src.getHitTestResults()[0];
if (hit) {
const pose = hit.getPose(referenceSpace);
// place a sphere at the hit location
}
});
}
renderer.xr.addEventListener('select', onSelect);
renderer.setAnimationLoop(() => renderer.render(scene, camera));
High-Performance Metal Rendering
For performance-critical applications, the macOS Spatial/Metal Engineer implements instanced rendering pipelines. The MetalGraphRenderer class manages GPU buffers and command encoding:
class MetalGraphRenderer {
private let device: MTLDevice
private let commandQueue: MTLCommandQueue
private var pipelineState: MTLRenderPipelineState
private var nodeBuffer: MTLBuffer // per-instance data
func render(nodes: [GraphNode], camera: Camera) {
guard let commandBuffer = commandQueue.makeCommandBuffer(),
let renderPass = view.currentRenderPassDescriptor,
let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPass) else { return }
// Update GPU buffers …
encoder.setRenderPipelineState(pipelineState)
encoder.setVertexBuffer(nodeBuffer, offset: 0, index: 0)
encoder.drawPrimitives(type: .triangleStrip,
vertexStart: 0,
vertexCount: 4,
instanceCount: nodes.count)
encoder.endEncoding()
commandBuffer.present(view.currentDrawable!)
commandBuffer.commit()
}
}
VisionOS Liquid-Glass Integration
The visionOS Spatial Engineer wraps 3D content in SwiftUI windows with RealityKit, applying Apple's Liquid-Glass design system through the glassBackgroundEffect modifier:
import SwiftUI
import RealityKit
struct ImmersiveView: View {
var body: some View {
WindowGroup {
RealityView { content in
// Load a 3D scene that contains the XR UI
let scene = try! await Entity.load(named: "XRScene")
content.add(scene)
}
.glassBackgroundEffect(.ultraThin) // Liquid-Glass styling
}
.defaultSize(width: 1024, height: 768)
}
}
Key Files and Agent Definitions
The spatial computing pipeline is defined across dedicated markdown files in the repository:
Summary
Spatial computing agents in the agency-agents repository form a modular, role-driven pipeline for immersive interface development:
- UX architects define ergonomic, motion-sickness-aware interfaces using spatial layout templates.
- Prototypers (WebXR and cockpit specialists) transform designs into interactive scenes with ray-casting and physics validation.
- Performance engineers (Metal and visionOS) replace generic renderers with native pipelines to achieve 90+ fps on large datasets.
- An orchestrator coordinates artifact hand-offs—passing UI specs, component libraries, and shaders between agents while tracking dependencies.
By leveraging dedicated markdown personality files and production-ready code patterns, teams can deploy immersive experiences across browsers, VisionOS, and high-performance native Metal applications.
Frequently Asked Questions
What are the primary roles of spatial computing agents in the agency-agents repository?
The spatial computing agents are organized into three primary layers: Interaction and UX (handled by the XR Interface Architect and XR Cockpit Interaction Specialist), Application Logic and Rendering (managed by the XR Immersive Developer and macOS Spatial/Metal Engineer), and Platform Integration (implemented by the visionOS Spatial Engineer). Each agent owns specific responsibilities ranging from spatial layout design to high-performance Metal pipeline optimization.
How do spatial computing agents handle performance optimization for complex 3D scenes?
For performance-critical workloads requiring sustained 90+ fps, the macOS Spatial/Metal Engineer replaces WebGL-based renderers with native Metal pipelines. As implemented in spatial-computing/macos-spatial-metal-engineer.md, the MetalGraphRenderer class utilizes instanced rendering via drawPrimitives(type:vertexStart:vertexCount:instanceCount:) and manages GPU buffers through MTLBuffer objects to efficiently render massive node graphs on macOS and Vision Pro.
What technologies do spatial computing agents use for WebXR prototyping?
The XR Immersive Developer leverages the WebXR Device API combined with Three.js or Babylon.js to scaffold cross-platform immersive applications. According to spatial-computing/xr-immersive-developer.md, typical implementations include XRButton for session management, requestHitTestSource for spatial ray-casting, and setAnimationLoop for render loops, enabling validation across Meta Quest, iOS browsers, and desktop environments.
How do spatial computing agents integrate with Apple VisionOS specifically?
The visionOS Spatial Engineer handles native integration through SwiftUI and RealityKit, as defined in spatial-computing/visionos-spatial-engineer.md. This agent wraps 3D content in WindowGroup containers, applies Apple's Liquid-Glass design system using the glassBackgroundEffect(.ultraThin) modifier, and manages CompositorServices for remote-immersive streaming when paired with Metal backends.
Have a question about this repo?
These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:
curl -s "https://instagit.com/install.md" 
Maintain an open-source project? Get it listed too →