# Implementing Context-Awareness for Agent-Native: How the Agent Knows the User's View

> Learn how to implement context awareness for agent native. Query user views via standardized memory APIs with real-time UI state and server-side sync.

- Repository: [Builder.io/agent-native](https://github.com/BuilderIO/agent-native)
- Tags: how-to-guide
- Published: 2026-06-27

---

**Agent-Native maintains a real-time sync between React UI state and server-side request context, allowing agents to query the user's current view through standardized memory APIs.**

When building intelligent agents in the BuilderIO/agent-native framework, the agent must understand what the user is currently seeing and interacting with. This repository implements a strict separation between **server-side request context** and **client-side UI context**, connected by a sync layer that keeps the agent's knowledge base constantly updated with the user's current view.

## Understanding the Context Architecture

The framework splits context management across two boundaries. **Server-side request context** handles authentication, workspace data, and view state during Nitro route execution. **Client-side UI context** manages React state for interactive elements like hovered components or active slides. A dedicated sync layer bridges these worlds, ensuring agents can access UI state within server actions without direct browser access.

## Server-Side Request Context

When a request enters a Nitro route (such as `/api/*` actions), the framework instantiates a request-context object containing the authenticated user, organization details, and active view information.

### Accessing User Identity and View State

The `@agent-native/core/server/request-context` package provides helper functions to extract context data within any action. These helpers read values attached by the `withRequestContext` middleware.

```ts
import { getRequestUserEmail, getRequestOrgId } from "@agent-native/core/server/request-context";

export const viewScreen = defineAction({
  name: "view-screen",
  async handler({ request }) {
    const { userId, orgId } = getRequestUserEmail(request);
    // Access the current screen from the request context
    await addToAgentMemory({ userId, orgId, screen: request.context?.screen });
  },
});

```

The implementation in [`packages/core/src/server/request-context.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/server/request-context.ts) exposes the context that travels with the request lifecycle, allowing actions to reference the user's current screen without manual parameter passing.

### The Request Context Middleware

The middleware automatically injects context into incoming requests. Actions defined with `defineAction` receive this context through the standard `request` object, making the user's current view available via `request.context?.screen` or other UI state properties.

## Client-Side UI Context in React

On the browser side, Agent-Native uses React Contexts to track user interactions. These contexts capture ephemeral UI state like mouse position, selected elements, and timeline positions.

### Tracking Interactive Elements

The `CurrentElementContext` defined in `templates/<template>/contexts/CurrentElementContext.ts` tracks which element the user is currently hovering over or editing.

```tsx
// templates/videos/contexts/CurrentElementContext.ts
import { createContext, useContext } from "react";

export type CurrentElement = { id: string; type: string };
export const CurrentElementContext = createContext<CurrentElement | null>(null);
export const useCurrentElement = () => useContext(CurrentElementContext);

```

The `useRegisterInteractiveElement` hook in [`templates/videos/app/remotion/hooks/useRegisterInteractiveElement.ts`](https://github.com/BuilderIO/agent-native/blob/main/templates/videos/app/remotion/hooks/useRegisterInteractiveElement.ts) automatically updates this context when users interact with components.

```tsx
import { useCurrentElement } from "@/contexts/CurrentElementContext";

export const useRegisterInteractiveElement = (element: CurrentElement) => {
  const setCurrent = useCurrentElement();
  return {
    onMouseEnter: () => setCurrent(element),
    onMouseLeave: () => setCurrent(null),
  };
};

```

### Using the CurrentElementContext Hook

Components that need to report their interactive status consume these hooks in their JSX. When a user hovers over a video clip, slide, or design component, the context updates immediately.

```tsx
function InteractiveVideoClip({ clip }) {
  const handlers = useRegisterInteractiveElement({ id: clip.id, type: "clip" });
  return <div {...handlers}>{clip.title}</div>;
}

```

Additional contexts like `CompositionContext` and `TimelineContext` track complex UI state including timeline positions and track arrangements, all accessible through standardized hooks such as `useComposition` and `useTimeline`.

## Syncing UI State to the Agent

The framework mirrors client UI context into the server request context through the `useDbSync` hook and a dedicated sync endpoint.

### The Sync Pipeline

Whenever React context updates on the client, the `syncContext` function in [`packages/core/src/client/sync.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/client/sync.ts) POSTs the current UI state to the `/_agent-native/sync-context` endpoint. This short-lived Nitro route:

1. Reads the current UI state (hovered elements, active slides, viewports)
2. Wraps the data in the request-context payload
3. Updates the agent's memory with the new view state

```tsx
import { useCurrentElement } from "@/contexts/CurrentElementContext";

export const HoverInfo = () => {
  const element = useCurrentElement();
  if (!element) return null;
  return <div className="tooltip">Hovering: {element.type} ({element.id})</div>;
};

```

### How Actions Receive Context

Once synced, any server action can access the UI context through the request object. The `view-screen` action in [`templates/videos/actions/view-screen.ts`](https://github.com/BuilderIO/agent-native/blob/main/templates/videos/actions/view-screen.ts) demonstrates this pattern by receiving the current screen name and updating agent memory accordingly.

```ts
import { getRequestUserEmail } from "@agent-native/core/server/request-context";

export const listUserDecks = defineAction({
  name: "list-decks",
  async handler({ request }) {
    const email = getRequestUserEmail(request);
    const view = request.context?.screen; // UI view injected by sync layer
    return db.select().from(decks).where(eq(decks.ownerEmail, email));
  },
});

```

Agents can now query this memory to understand the user's current context before generating responses.

## Practical Implementation Examples

### Creating Interactive Components

To make a component report its state to the agent, wrap it with the registration hook.

```tsx
import { useRegisterInteractiveElement } from "@/app/remotion/hooks/useRegisterInteractiveElement";

export const InteractiveButton = ({ id }) => {
  const handlers = useRegisterInteractiveElement({ id, type: "button" });
  return <button {...handlers}>Click me</button>;
};

```

### Building Context-Aware Actions

Actions can reference the current view to tailor responses. Access the request context through the standard handler payload.

```ts
await agent.run(
  `The user is currently looking at screen "${await getAgentMemory('screen')}". ` +
  `Show them the next slide.`
);

```

### Extending with Custom Context

To expose additional UI state like zoom levels:

1. Create a new React context in your template's `contexts/` folder.
2. Update [`packages/core/src/client/sync.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/client/sync.ts) to include the value in the sync payload.
3. Access it in actions via `request.context?.zoomLevel`.

**Example implementation:**

```tsx
// templates/videos/contexts/ZoomContext.ts
import { createContext, useContext } from "react";
export const ZoomContext = createContext<number>(1);
export const useZoom = () => useContext(ZoomContext);

```

```ts
// packages/core/src/client/sync.ts
export const syncContext = async () => {
  const zoom = useZoom();
  await fetch("/_agent-native/sync-context", {
    method: "POST",
    body: JSON.stringify({ zoomLevel: zoom }),
  });
};

```

```ts
// Server action
export const zoomInfo = defineAction({
  name: "zoom-info",
  async handler({ request }) {
    const zoom = request.context?.zoomLevel ?? 1;
    return `Current zoom is ${zoom * 100}%`;
  },
});

```

## Summary

- **Agent-Native separates concerns** between server-side request context (authentication, persistent state) and client-side UI context (React interactions, ephemeral state).
- **The sync layer** in [`packages/core/src/client/sync.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/client/sync.ts) automatically mirrors React context values to the server via `/_agent-native/sync-context`.
- **Interactive components** use `useRegisterInteractiveElement` to report hover and selection states to `CurrentElementContext`.
- **Server actions** access UI state through `request.context` after importing helpers from `@agent-native/core/server/request-context`.
- **Extending context** requires only adding a React context, updating the sync payload, and reading the value in actions—no boilerplate changes needed.

## Frequently Asked Questions

### How does the agent access the current screen name in Agent-Native?

The agent accesses the current screen through the request context object. When a user navigates, the client syncs the screen name to the server via the `/_agent-native/sync-context` endpoint. Server actions then read this value from `request.context?.screen` and can store it in agent memory using `addToAgentMemory`. This allows subsequent agent runs to reference the exact view the user is seeing.

### What is the difference between CurrentElementContext and CompositionContext?

`CurrentElementContext` tracks the specific DOM element or component the user is currently hovering over or editing, providing granular interaction data like element IDs and types. `CompositionContext` and `TimelineContext` manage higher-level application state such as timeline positions, track arrangements, and composition properties. Both contexts sync to the server, but they serve different granularity levels of UI awareness.

### Can I add custom context fields without modifying the core framework?

Yes. You can extend context-awareness by creating new React contexts in your template's `contexts/` directory, updating [`packages/core/src/client/sync.ts`](https://github.com/BuilderIO/agent-native/blob/main/packages/core/src/client/sync.ts) to include the new values in the sync payload, and accessing them in actions via `request.context?.yourField`. The framework automatically propagates these custom fields through the sync pipeline without requiring changes to the core server logic.

### Why does Agent-Native use a sync endpoint instead of WebSockets?

The sync endpoint (`/_agent-native/sync-context`) provides a simple, stateless mechanism to push UI context to the server that integrates cleanly with Nitro's request-response model. This approach aligns with the framework's serverless-friendly architecture while ensuring that any action invocation has access to the most recent UI state through the standard request context, without maintaining persistent connections.