Implementing Context-Awareness for Agent-Native: How the Agent Knows the User's View

Agent-Native maintains a real-time sync between React UI state and server-side request context, allowing agents to query the user's current view through standardized memory APIs.

When building intelligent agents in the BuilderIO/agent-native framework, the agent must understand what the user is currently seeing and interacting with. This repository implements a strict separation between server-side request context and client-side UI context, connected by a sync layer that keeps the agent's knowledge base constantly updated with the user's current view.

Understanding the Context Architecture

The framework splits context management across two boundaries. Server-side request context handles authentication, workspace data, and view state during Nitro route execution. Client-side UI context manages React state for interactive elements like hovered components or active slides. A dedicated sync layer bridges these worlds, ensuring agents can access UI state within server actions without direct browser access.

Server-Side Request Context

When a request enters a Nitro route (such as /api/* actions), the framework instantiates a request-context object containing the authenticated user, organization details, and active view information.

Accessing User Identity and View State

The @agent-native/core/server/request-context package provides helper functions to extract context data within any action. These helpers read values attached by the withRequestContext middleware.

import { getRequestUserEmail, getRequestOrgId } from "@agent-native/core/server/request-context";

export const viewScreen = defineAction({
  name: "view-screen",
  async handler({ request }) {
    const { userId, orgId } = getRequestUserEmail(request);
    // Access the current screen from the request context
    await addToAgentMemory({ userId, orgId, screen: request.context?.screen });
  },
});

The implementation in packages/core/src/server/request-context.ts exposes the context that travels with the request lifecycle, allowing actions to reference the user's current screen without manual parameter passing.

The Request Context Middleware

The middleware automatically injects context into incoming requests. Actions defined with defineAction receive this context through the standard request object, making the user's current view available via request.context?.screen or other UI state properties.

Client-Side UI Context in React

On the browser side, Agent-Native uses React Contexts to track user interactions. These contexts capture ephemeral UI state like mouse position, selected elements, and timeline positions.

Tracking Interactive Elements

The CurrentElementContext defined in templates/<template>/contexts/CurrentElementContext.ts tracks which element the user is currently hovering over or editing.

// templates/videos/contexts/CurrentElementContext.ts
import { createContext, useContext } from "react";

export type CurrentElement = { id: string; type: string };
export const CurrentElementContext = createContext<CurrentElement | null>(null);
export const useCurrentElement = () => useContext(CurrentElementContext);

The useRegisterInteractiveElement hook in templates/videos/app/remotion/hooks/useRegisterInteractiveElement.ts automatically updates this context when users interact with components.

import { useCurrentElement } from "@/contexts/CurrentElementContext";

export const useRegisterInteractiveElement = (element: CurrentElement) => {
  const setCurrent = useCurrentElement();
  return {
    onMouseEnter: () => setCurrent(element),
    onMouseLeave: () => setCurrent(null),
  };
};

Using the CurrentElementContext Hook

Components that need to report their interactive status consume these hooks in their JSX. When a user hovers over a video clip, slide, or design component, the context updates immediately.

function InteractiveVideoClip({ clip }) {
  const handlers = useRegisterInteractiveElement({ id: clip.id, type: "clip" });
  return <div {...handlers}>{clip.title}</div>;
}

Additional contexts like CompositionContext and TimelineContext track complex UI state including timeline positions and track arrangements, all accessible through standardized hooks such as useComposition and useTimeline.

Syncing UI State to the Agent

The framework mirrors client UI context into the server request context through the useDbSync hook and a dedicated sync endpoint.

The Sync Pipeline

Whenever React context updates on the client, the syncContext function in packages/core/src/client/sync.ts POSTs the current UI state to the /_agent-native/sync-context endpoint. This short-lived Nitro route:

1. Reads the current UI state (hovered elements, active slides, viewports)
2. Wraps the data in the request-context payload
3. Updates the agent's memory with the new view state

import { useCurrentElement } from "@/contexts/CurrentElementContext";

export const HoverInfo = () => {
  const element = useCurrentElement();
  if (!element) return null;
  return <div className="tooltip">Hovering: {element.type} ({element.id})</div>;
};

How Actions Receive Context

Once synced, any server action can access the UI context through the request object. The view-screen action in templates/videos/actions/view-screen.ts demonstrates this pattern by receiving the current screen name and updating agent memory accordingly.

import { getRequestUserEmail } from "@agent-native/core/server/request-context";

export const listUserDecks = defineAction({
  name: "list-decks",
  async handler({ request }) {
    const email = getRequestUserEmail(request);
    const view = request.context?.screen; // UI view injected by sync layer
    return db.select().from(decks).where(eq(decks.ownerEmail, email));
  },
});

Agents can now query this memory to understand the user's current context before generating responses.

Practical Implementation Examples

Creating Interactive Components

To make a component report its state to the agent, wrap it with the registration hook.

import { useRegisterInteractiveElement } from "@/app/remotion/hooks/useRegisterInteractiveElement";

export const InteractiveButton = ({ id }) => {
  const handlers = useRegisterInteractiveElement({ id, type: "button" });
  return <button {...handlers}>Click me</button>;
};

Building Context-Aware Actions

Actions can reference the current view to tailor responses. Access the request context through the standard handler payload.

await agent.run(
  `The user is currently looking at screen "${await getAgentMemory('screen')}". ` +
  `Show them the next slide.`
);

Extending with Custom Context

To expose additional UI state like zoom levels:

1. Create a new React context in your template's contexts/ folder.
2. Update packages/core/src/client/sync.ts to include the value in the sync payload.
3. Access it in actions via request.context?.zoomLevel.

Example implementation:

// templates/videos/contexts/ZoomContext.ts
import { createContext, useContext } from "react";
export const ZoomContext = createContext<number>(1);
export const useZoom = () => useContext(ZoomContext);

// packages/core/src/client/sync.ts
export const syncContext = async () => {
  const zoom = useZoom();
  await fetch("/_agent-native/sync-context", {
    method: "POST",
    body: JSON.stringify({ zoomLevel: zoom }),
  });
};

// Server action
export const zoomInfo = defineAction({
  name: "zoom-info",
  async handler({ request }) {
    const zoom = request.context?.zoomLevel ?? 1;
    return `Current zoom is ${zoom * 100}%`;
  },
});

Summary

• Agent-Native separates concerns between server-side request context (authentication, persistent state) and client-side UI context (React interactions, ephemeral state).
• The sync layer in packages/core/src/client/sync.ts automatically mirrors React context values to the server via /_agent-native/sync-context.
• Interactive components use useRegisterInteractiveElement to report hover and selection states to CurrentElementContext.
• Server actions access UI state through request.context after importing helpers from @agent-native/core/server/request-context.
• Extending context requires only adding a React context, updating the sync payload, and reading the value in actions—no boilerplate changes needed.

Frequently Asked Questions

How does the agent access the current screen name in Agent-Native?

The agent accesses the current screen through the request context object. When a user navigates, the client syncs the screen name to the server via the /_agent-native/sync-context endpoint. Server actions then read this value from request.context?.screen and can store it in agent memory using addToAgentMemory. This allows subsequent agent runs to reference the exact view the user is seeing.

What is the difference between CurrentElementContext and CompositionContext?

CurrentElementContext tracks the specific DOM element or component the user is currently hovering over or editing, providing granular interaction data like element IDs and types. CompositionContext and TimelineContext manage higher-level application state such as timeline positions, track arrangements, and composition properties. Both contexts sync to the server, but they serve different granularity levels of UI awareness.

Can I add custom context fields without modifying the core framework?

Yes. You can extend context-awareness by creating new React contexts in your template's contexts/ directory, updating packages/core/src/client/sync.ts to include the new values in the sync payload, and accessing them in actions via request.context?.yourField. The framework automatically propagates these custom fields through the sync pipeline without requiring changes to the core server logic.

Why does Agent-Native use a sync endpoint instead of WebSockets?

The sync endpoint (/_agent-native/sync-context) provides a simple, stateless mechanism to push UI context to the server that integrates cleanly with Nitro's request-response model. This approach aligns with the framework's serverless-friendly architecture while ensuring that any action invocation has access to the most recent UI state through the standard request context, without maintaining persistent connections.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}