How the Annotation Engine Resolves Voice Annotations to World-Anchored Entities in God's Eye View

The annotation engine resolves voice annotations to world-anchored entities through a three-stage pipeline: parsing voice commands into annotation specs, resolving geographic coordinates via resolveAnnotationTarget, and rendering persistent Cesium entities that remain fixed to real-world coordinates regardless of camera movement.

The annotation engine serves as the core "whiteboard" subsystem in the bilawalsidhu/gods-eye-view repository, transforming spoken place references into persistent 3-D visual marks. Understanding how the annotation engine resolves voice annotations to world-anchored entities requires examining the interaction between the voice layer, the resolution logic, and the Cesium-based rendering pipeline.

The Three-Stage Resolution Pipeline

The resolution flow operates through three distinct stages that convert speech into persistent geographic markers:

1. Voice → Action: The GEVi voice layer parses utterances and creates tool calls containing annotation specs
2. Spec → World Anchor: The engine resolves place names into latitude/longitude coordinates and initiates deferred polygon queries
3. World Anchor → Renderer: The system creates Cesium entities anchored to geographic coordinates and optionally upgrades them with footprint data

Stage 1: Voice Input to Annotation Specs

The process begins in src/voice/gevRealtime.js, where the GEVi voice agent processes incoming utterances. When a user references a location, the voice layer generates a tool call named annotate_map containing one or more annotation specs.

Each spec includes:

• type: The annotation category (e.g., pin, area, compound)
• target: The spoken place name or coordinates
• label: Optional display text
• color: Visual styling hint
• footprint: Boolean flag requesting polygon outlines

// Voice layer forwarding specs to the engine
const result = await annotations.annotate(requests, {
  flyTo: args.flyTo, 
  persist: args.persist, 
  clearPrevious: args.clearPrevious,
});

The annotations object here is the initialized engine instance created via initAnnotations() in src/main.js, which wires the voice subsystem to the annotation API.

Stage 2: Resolving Geographic Anchors and Footprints

Once the engine receives the specs via annotationEngine.annotate(), it processes each spec concurrently through resolveSpec(). This function invokes resolveAnnotationTarget from src/annotations/annotationResolver.js to perform the actual geographic resolution.

Geocoding and Coordinate Lookup

The resolveAnnotationTarget function accepts a target string and optional coordinates, then performs the following resolution logic:

const r = await resolveTarget({
  viewer,
  target: name,
  latitude, longitude,
  screenX, screenY,
  footprint: wantFootprint,
  deferFootprint: wantFootprint,
  signal,
});

The resolver first attempts a Places service lookup to convert the spoken name into a geographic anchor. If the Places service returns a match, it yields a normalized anchor object containing lon, lat, and height values. If no match exists, the engine falls back to any explicit coordinates provided in the voice command.

Deferred Overpass Queries for Polygons

When the spec requests a footprint (area, compound, etc.), the resolver initiates a deferred Overpass API query before returning the initial anchor. This allows the engine to display a point marker immediately while fetching the complex polygon geometry asynchronously.

The deferred query stores a promise that later resolves to a ring (polygon coordinates) and footprintKind metadata. This design prevents voice interaction latency while ensuring the annotation eventually receives its full geographic boundary.

Stage 3: World-Anchored Rendering and Outline Upgrades

With geographic coordinates established, the engine transitions to visualization through the pluggable renderer system.

Creating Runtime Annotation Objects

The buildAnnotation() function (lines 686-748 in src/annotations/annotationEngine.js) constructs the canonical runtime annotation object:

// Simplified structure created by buildAnnotation()
{
  anchor: { lon, lat, height },
  targetKey: normalizedName,  // For deduplication
  type: 'area',
  ring: null,  // Populated later by outline upgrade
  // ... visual properties
}

The anchor field is critical—it stores the world-anchored coordinates that remain fixed to the globe regardless of camera position. The engine maintains these objects in an internal Map called liveAnnotations to track state and enforce lifecycle rules.

Rendering via Cesium Entities

The engine passes completed annotation objects to either worldAnnotationRenderer.js (for globe-anchored marks) or screenAnnotationRenderer.js (for screen-space overlays). The world-space renderer creates native Cesium Entity instances attached to a CustomDataSource:

// World-space renderer creates globe-anchored entities
renderer.add(anno);
// Creates Cesium Entity with position: Cartesian3.fromDegrees(anchor.lon, anchor.lat, anchor.height)

Because these entities use Cartesian3 positions derived from the geographic anchor, they automatically maintain their location relative to the Earth's surface as the user pans, zooms, or tilts the camera.

Deferred Outline Enhancement

If the original spec requested a footprint, the engine queues a background task via startOutlineUpgrade(). The runOutlineUpgrade() function calls resolveOutlineWithRetry() (which handles transient Overpass failures with exponential backoff) to fetch the polygon data.

Once retrieved, the engine mutates the existing annotation in-place:

anno.ring = fp.ring;
anno.footprintKind = fp.kind;
renderer.update(anno);  // Visual morph from point to polygon

This in-place mutation ensures the annotation remains world-anchored while upgrading its visual representation from a simple pin to a full area overlay.

Lifecycle Management and Deduplication

The engine enforces strict resource limits and data integrity through mechanisms defined in annotationEngine.js (lines 1026-1084):

• Hard Cap: Maximum of MAX_LIVE_ANNOTATIONS = 120 concurrent marks
• TTL Fading: Annotations fade out after a time-to-live period unless marked with persist: true
• Geometric Deduplication: The engine compares anchor coordinates and ring geometry to prevent stacked duplicates of the same geographic feature

All lifecycle operations preserve the world-anchored nature of the entities, ensuring visual consistency across navigation sessions.

Implementation Example

The following patterns demonstrate how to interact with the resolution pipeline programmatically:

// Initialize the engine (normally performed in main.js)
import { initAnnotations } from './annotations/index.js';
const annotations = initAnnotations({ viewer, tileset });

// Direct annotation from code (mirrors voice layer behavior)
await annotations.annotate([
  {
    type: 'area',
    target: 'Presidio of San Francisco',
    label: 'The Presidio (former Army base)',
    color: 'green',
    footprint: true,          // Request polygon outline via Overpass
  },
  {
    type: 'pin',
    target: 'Letterman Digital Arts Center, San Francisco',
    label: 'ILM / Lucasfilm',
    color: 'cyan',
  },
], { flyTo: true, persist: true });

Monitor asynchronous outline completion:

const unsubscribe = annotations.onOutlineEvent(evt => {
  console.log(`Outline ${evt.status} for ${evt.label || evt.target}`);
  // 'success', 'failed', or 'timeout'
});
// Cleanup: unsubscribe();

Run built-in demonstrations:

await annotations.demo();   // Short San Francisco tour
await annotations.tour();   // Full scripted camera movement

Summary

• The annotation engine in bilawalsidhu/gods-eye-view converts voice commands into world-anchored entities through a three-stage pipeline involving gevRealtime.js, annotationResolver.js, and worldAnnotationRenderer.js.
• Geographic resolution occurs via resolveAnnotationTarget, which queries the Places service for coordinates and optionally defers Overpass queries for polygon footprints.
• Runtime annotation objects store fixed anchor coordinates (lon/lat/height) that bind visual marks to specific Earth locations, independent of camera perspective.
• The deferred upgrade system fetches complex geometries asynchronously via resolveOutlineWithRetry, updating existing annotations in-place without breaking world anchoring.
• Lifecycle management enforces a 120-annotation limit, TTL-based fading, and geometric deduplication to maintain performance and prevent duplicate marks.

Frequently Asked Questions

How does the engine handle ambiguous place names from voice input?

The engine relies on the Places service geocoding in src/annotations/annotationResolver.js to disambiguate spoken names. If multiple matches exist, the service typically returns the most prominent result. The resolver also accepts explicit latitude and longitude parameters in the annotation spec as fallback coordinates when the Places service returns no matches or when the voice command includes specific coordinate data.

What happens if the Overpass query for a footprint fails?

The resolveOutlineWithRetry function implements exponential backoff for transient failures. If the query ultimately fails after retries, the annotation remains visible as a point marker (pin) rather than an area, and the engine emits an onOutlineEvent with status 'failed' or 'timeout'. The world-anchored point persists indefinitely if persist: true was set, ensuring the user still sees the reference location even without the polygon boundary.

How does the engine prevent duplicate annotations for the same location?

The engine maintains a targetKey derived from normalized place names and compares both anchor coordinates (lon/lat) and ring geometries before adding new marks. According to the deduplication logic in annotationEngine.js (lines 1026-1084), if a new request matches an existing annotation's anchor within tolerance and shares the same footprint geometry, the engine rejects the duplicate rather than stacking multiple markers at the same geographic position.

Can annotations persist across camera movements and scene changes?

Yes. Because annotations are world-anchored to geographic coordinates (lon/lat/height) rather than screen positions, they automatically persist across all camera movements, zoom levels, and orbital changes. The Cesium Entity instances created by worldAnnotationRenderer.js remain attached to the globe's surface. For session persistence, the persist: true option in the annotation spec prevents TTL-based fading, keeping marks alive indefinitely until explicitly cleared via clearPrevious or manual removal.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}