# How Spatial Computing Agents Handle Immersive Interface Development: A Three-Layer Pipeline

> Discover how spatial computing agents in agency-agents handle immersive interface development using a three-layer pipeline for cross-platform UX, WebXR, and native rendering.

- Repository: [Michael Sitarzewski/agency-agents](https://github.com/msitarzewski/agency-agents)
- Tags: how-to-guide
- Published: 2026-03-09

---

**Spatial computing agents in the msitarzewski/agency-agents repository employ a modular, role-driven pipeline where specialized agents handle UX architecture, WebXR prototyping, and native Metal or VisionOS rendering to deliver cross-platform immersive experiences.**

The spatial computing agents form a comprehensive architecture that covers everything from ergonomic 3D interface design to high-performance native rendering. By combining specialized expertise across interaction design, application logic, and platform integration, these agents enable the development of immersive interfaces that run in browsers, on VisionOS, or as Metal-based renderers on macOS and Apple Vision Pro.

## The Three-Layer Architecture of Spatial Computing Agents

The repository organizes spatial computing agents into three distinct layers, each owning a specific slice of the immersive-interface stack.

### Interaction and UX Layer

The **XR Interface Architect** and **XR Cockpit Interaction Specialist** handle spatial layout and ergonomic design. According to [`spatial-computing/xr-interface-architect.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-interface-architect.md), the architect drafts comfortable 3D UI flows, defines interaction zones, and produces layout templates that mitigate motion sickness. The cockpit specialist, documented in [`spatial-computing/xr-cockpit-interaction-specialist.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-cockpit-interaction-specialist.md), extends these designs with fixed-perspective controls such as yokes, levers, and gauges, encoding ergonomic rules for multimodal input patterns including gaze, pinch, and hand-tracking.

### Application Logic and Rendering Layer

The **XR Immersive Developer** and **macOS Spatial/Metal Engineer** manage core scene graphs and performance-critical rendering. As implemented in [`spatial-computing/xr-immersive-developer.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-immersive-developer.md), the immersive developer scaffolds cross-platform WebXR projects using Three.js or Babylon.js, wiring ray-casting, hit-testing, and physics validation. For workloads requiring sustained performance above 90 fps on massive datasets, the macOS Spatial/Metal Engineer—defined in [`spatial-computing/macos-spatial-metal-engineer.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/macos-spatial-metal-engineer.md)—replaces WebGL renderers with native Metal pipelines, streaming frames to Vision Pro via `RemoteImmersiveSpace`.

### Platform Integration Layer

The **visionOS Spatial Engineer** handles native Apple ecosystem integration. According to [`spatial-computing/visionos-spatial-engineer.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/visionos-spatial-engineer.md), this agent wraps scenes in SwiftUI `WindowGroup` containers, applies Apple's Liquid-Glass design system through modifiers like `glassBackgroundEffect(.ultraThin)`, and implements accessibility hooks for VoiceOver and Switch Control. The engineer also manages `CompositorServices` for remote-immersive streaming when paired with Metal backends.

## Architectural Flow: From Concept to Deployment

The spatial computing agents follow a seven-stage pipeline that transforms experience concepts into deployable immersive applications.

1. **Concept and UX Definition** – The XR Interface Architect drafts UI flows and produces layout templates stored as reusable components.

2. **Interaction Specification** – The XR Cockpit Interaction Specialist extends designs with cockpit-style controls and encodes ergonomic constraints for spatial input.

3. **Web-Based Prototyping** – The XR Immersive Developer scaffolds a WebXR project in [`src/main.js`](https://github.com/msitarzewski/agency-agents/blob/main/src/main.js), implementing `requestHitTestSource` for ray-casting and validating the experience across Meta Quest, iOS browsers, and desktop.

4. **High-Performance Native Build** – For demanding workloads, the macOS Spatial/Metal Engineer implements a `MetalGraphRenderer` class with instanced rendering, replacing the WebGL pipeline to achieve 90+ fps on large node graphs.

5. **VisionOS Integration** – The visionOS Spatial Engineer wraps the scene in a `RealityView`, applies `glassBackgroundEffect(.ultraThin)` for Liquid-Glass styling, and manages `WindowGroup` presentation.

6. **Orchestration** – The Agents Orchestrator ([`specialized/agents-orchestrator.md`](https://github.com/msitarzewski/agency-agents/blob/main/specialized/agents-orchestrator.md)) coordinates hand-offs, passing artifacts such as UI specs, component libraries, and shaders between agents while tracking dependencies.

7. **Deployment and Monitoring** – Final bundles deploy as WebXR sites, VisionOS apps, or macOS Metal applications, tracked through strategy runbooks ([`strategy/playbooks/phase-3-build.md`](https://github.com/msitarzewski/agency-agents/blob/main/strategy/playbooks/phase-3-build.md)).

## Implementation Examples

### WebXR Prototyping with Three.js

The XR Immersive Developer creates entry points using the WebXR Device API with Three.js. The `onSelect` handler demonstrates hit-testing implementation:

```javascript
// src/main.js – minimal WebXR entry point
import { XRButton } from 'three/examples/jsm/webxr/XRButton.js';
import { Scene, PerspectiveCamera, WebGLRenderer } from 'three';

const scene = new Scene();
const camera = new PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 100);
const renderer = new WebGLRenderer({ antialias: true });
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.xr.enabled = true;
document.body.appendChild(renderer.domElement);
document.body.appendChild(XRButton.createButton(renderer));

// Simple hit-test raycast
function onSelect(event) {
  const referenceSpace = renderer.xr.getReferenceSpace();
  const session = renderer.xr.getSession();
  session.requestHitTestSource({ space: referenceSpace }).then(src => {
    const hit = src.getHitTestResults()[0];
    if (hit) {
      const pose = hit.getPose(referenceSpace);
      // place a sphere at the hit location
    }
  });
}
renderer.xr.addEventListener('select', onSelect);
renderer.setAnimationLoop(() => renderer.render(scene, camera));

```

### High-Performance Metal Rendering

For performance-critical applications, the macOS Spatial/Metal Engineer implements instanced rendering pipelines. The `MetalGraphRenderer` class manages GPU buffers and command encoding:

```swift
class MetalGraphRenderer {
    private let device: MTLDevice
    private let commandQueue: MTLCommandQueue
    private var pipelineState: MTLRenderPipelineState
    private var nodeBuffer: MTLBuffer   // per-instance data

    func render(nodes: [GraphNode], camera: Camera) {
        guard let commandBuffer = commandQueue.makeCommandBuffer(),
              let renderPass = view.currentRenderPassDescriptor,
              let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPass) else { return }

        // Update GPU buffers …
        encoder.setRenderPipelineState(pipelineState)
        encoder.setVertexBuffer(nodeBuffer, offset: 0, index: 0)
        encoder.drawPrimitives(type: .triangleStrip,
                               vertexStart: 0,
                               vertexCount: 4,
                               instanceCount: nodes.count)
        encoder.endEncoding()
        commandBuffer.present(view.currentDrawable!)
        commandBuffer.commit()
    }
}

```

### VisionOS Liquid-Glass Integration

The visionOS Spatial Engineer wraps 3D content in SwiftUI windows with RealityKit, applying Apple's Liquid-Glass design system through the `glassBackgroundEffect` modifier:

```swift
import SwiftUI
import RealityKit

struct ImmersiveView: View {
    var body: some View {
        WindowGroup {
            RealityView { content in
                // Load a 3D scene that contains the XR UI
                let scene = try! await Entity.load(named: "XRScene")
                content.add(scene)
            }
            .glassBackgroundEffect(.ultraThin)   // Liquid-Glass styling
        }
        .defaultSize(width: 1024, height: 768)
    }
}

```

## Key Files and Agent Definitions

The spatial computing pipeline is defined across dedicated markdown files in the repository:

| File | Agent Role |
|------|------------|
| [[`spatial-computing/xr-interface-architect.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-interface-architect.md)](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-interface-architect.md) | Designs ergonomic 3D UI flows and spatial layouts |
| [[`spatial-computing/xr-cockpit-interaction-specialist.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-cockpit-interaction-specialist.md)](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-cockpit-interaction-specialist.md) | Implements fixed-perspective cockpit controls |
| [[`spatial-computing/xr-immersive-developer.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-immersive-developer.md)](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-immersive-developer.md) | Builds cross-platform WebXR applications |
| [[`spatial-computing/macos-spatial-metal-engineer.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/macos-spatial-metal-engineer.md)](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/macos-spatial-metal-engineer.md) | Optimizes performance via native Metal pipelines |
| [[`spatial-computing/visionos-spatial-engineer.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/visionos-spatial-engineer.md)](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/visionos-spatial-engineer.md) | Integrates with VisionOS SwiftUI and RealityKit |
| [[`specialized/agents-orchestrator.md`](https://github.com/msitarzewski/agency-agents/blob/main/specialized/agents-orchestrator.md)](https://github.com/msitarzewski/agency-agents/blob/main/specialized/agents-orchestrator.md) | Coordinates hand-offs between agents |
| [[`examples/nexus-spatial-discovery.md`](https://github.com/msitarzewski/agency-agents/blob/main/examples/nexus-spatial-discovery.md)](https://github.com/msitarzewski/agency-agents/blob/main/examples/nexus-spatial-discovery.md) | End-to-end demo of 3D AI-agent command center |

## Summary

Spatial computing agents in the agency-agents repository form a **modular, role-driven pipeline** for immersive interface development:

- **UX architects** define ergonomic, motion-sickness-aware interfaces using spatial layout templates.
- **Prototypers** (WebXR and cockpit specialists) transform designs into interactive scenes with ray-casting and physics validation.
- **Performance engineers** (Metal and visionOS) replace generic renderers with native pipelines to achieve 90+ fps on large datasets.
- An **orchestrator** coordinates artifact hand-offs—passing UI specs, component libraries, and shaders between agents while tracking dependencies.

By leveraging dedicated markdown personality files and production-ready code patterns, teams can deploy immersive experiences across browsers, VisionOS, and high-performance native Metal applications.

## Frequently Asked Questions

### What are the primary roles of spatial computing agents in the agency-agents repository?

The spatial computing agents are organized into three primary layers: **Interaction and UX** (handled by the XR Interface Architect and XR Cockpit Interaction Specialist), **Application Logic and Rendering** (managed by the XR Immersive Developer and macOS Spatial/Metal Engineer), and **Platform Integration** (implemented by the visionOS Spatial Engineer). Each agent owns specific responsibilities ranging from spatial layout design to high-performance Metal pipeline optimization.

### How do spatial computing agents handle performance optimization for complex 3D scenes?

For performance-critical workloads requiring sustained 90+ fps, the **macOS Spatial/Metal Engineer** replaces WebGL-based renderers with native Metal pipelines. As implemented in [`spatial-computing/macos-spatial-metal-engineer.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/macos-spatial-metal-engineer.md), the `MetalGraphRenderer` class utilizes instanced rendering via `drawPrimitives(type:vertexStart:vertexCount:instanceCount:)` and manages GPU buffers through `MTLBuffer` objects to efficiently render massive node graphs on macOS and Vision Pro.

### What technologies do spatial computing agents use for WebXR prototyping?

The **XR Immersive Developer** leverages the **WebXR Device API** combined with **Three.js** or Babylon.js to scaffold cross-platform immersive applications. According to [`spatial-computing/xr-immersive-developer.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/xr-immersive-developer.md), typical implementations include `XRButton` for session management, `requestHitTestSource` for spatial ray-casting, and `setAnimationLoop` for render loops, enabling validation across Meta Quest, iOS browsers, and desktop environments.

### How do spatial computing agents integrate with Apple VisionOS specifically?

The **visionOS Spatial Engineer** handles native integration through **SwiftUI** and **RealityKit**, as defined in [`spatial-computing/visionos-spatial-engineer.md`](https://github.com/msitarzewski/agency-agents/blob/main/spatial-computing/visionos-spatial-engineer.md). This agent wraps 3D content in `WindowGroup` containers, applies Apple's **Liquid-Glass** design system using the `glassBackgroundEffect(.ultraThin)` modifier, and manages `CompositorServices` for remote-immersive streaming when paired with Metal backends.