# Observation Space vs Action Space in LLM Agents: Key Differences Explained

> Understand the key differences between observation space and action space in LLM agents. Learn what information agents perceive and what actions they can perform.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: deep-dive
- Published: 2026-08-05

---

**The observation space defines all information an LLM agent can perceive from its environment, while the action space defines all operations the agent is permitted to execute on the external world.**

Understanding the distinction between observation space and action space is fundamental to designing effective AI agent systems. According to the `bojieli/ai-agent-book` repository, these two spaces form the complete interface between a frozen language model and its operating environment, determining what the agent knows and what it can actually accomplish.

## What Is the Observation Space?

The **observation space** encompasses every data source that feeds context into the LLM at decision time. As defined in [`book-en/chapter1.md`](https://github.com/bojieli/ai-agent-book/blob/main/book-en/chapter1.md), this space "translates information in the environment into context the model can process."

Key components of a typical observation space include:

- **UI state representations** such as HTML DOM trees and base64-encoded screenshots
- **Tool outputs** including search results, API responses, and database query results
- **Internal memory** and knowledge bases maintained across conversation turns
- **Task-specific context** provided by the user or system prompt

In the reference implementation found in [`chapter8/gaia-experience/AWorld/examples/phone_use/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/gaia-experience/AWorld/examples/phone_use/agent.py), the observation arrives as a structured object:

```python
def policy(self, observation: Observation, info: Dict[str, Any] = {}, **kwargs) -> List[ActionModel]:
    # observation.content holds the textual description

    # observation.dom_tree holds the parsed HTML

    # observation.image holds a screenshot

    xml_content, base64_img = observation.dom_tree, observation.image
    # Processing logic determines the next action...

```

If the observation space is too narrow, the model may lack essential data and fail to reason correctly, even if it possesses the necessary knowledge internally.

## What Is the Action Space?

The **action space** describes the complete set of operations the agent is authorized to perform. The `bojieli/ai-agent-book` explains that this space "translates model decisions into operations on the outside world." While the observation space expands what the model knows, the action space expands what the model can actually do.

Common action space components include:

- **Tool calls** such as web searches, code execution environments, and file manipulation
- **System commands** including run, stop, copy, and process management
- **Message sending** capabilities for chat, email, or API notifications
- **Direct API invocations** to external services

A limited action space forces the model to only suggest actions in natural language, even when it knows the exact technical steps required. The same [`agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/agent.py) file demonstrates how actions are structured for execution:

```python
def _make_history_item(self, llm_result, observation, metadata):
    # Build an action record that the execution layer will run

    action = {
        "type": "click",
        "target": llm_result["click_target"],
        "metadata": metadata,
    }
    # The framework translates this dict into an actual UI click

```

## Key Differences and Design Implications

When architecting LLM agents, the distinction between these spaces determines your systems-engineering leverage points:

**Observation Space** acts as the input boundary. Redefining or enlarging this space is often the primary lever for improving performance when the underlying model is fixed. For example, the book notes that "Manus enlarged the observation space with a virtual browser" to give the model richer visual context.

**Action Space** acts as the output boundary. Adding new tools or expanding existing ones increases an agent's capabilities without requiring model retraining or fine-tuning. The Manus example continues: "its file system, code execution, and command execution enlarged the action space."

The relationship is complementary: a richer observation space enables more informed reasoning, while a richer action space enables that reasoning to produce more effective outcomes.

## Practical Implementation: The Observation-Reasoning-Action Loop

A well-designed agent continuously cycles through three phases. First, it **observes** by reading the current state through the observation space. Second, it **reasons** by processing that context through the LLM. Third, it **acts** by issuing commands mapped to its action space.

This loop is evident in the `policy` method from [`chapter8/gaia-experience/AWorld/examples/phone_use/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/gaia-experience/AWorld/examples/phone_use/agent.py), which accepts an `Observation` object and returns a `List[ActionModel]`. The method signature itself represents the boundary between the two spaces: `Observation` flows in, `ActionModel` flows out.

As noted in [`book-en/reference-answers.md`](https://github.com/bojieli/ai-agent-book/blob/main/book-en/reference-answers.md), troubleshooting often involves determining which space to expand. If the agent demonstrates ignorance of available information, expand the observation space. If the agent understands what to do but cannot execute, expand the action space.

## Summary

- The **observation space** defines what the LLM can perceive (inputs), including UI states, tool outputs, and memory, while the **action space** defines what the LLM can execute (outputs), including tool calls and system commands.
- Expanding the observation space improves the model's contextual awareness without changing the model weights, making it the primary lever when the agent lacks necessary information.
- Expanding the action space unlocks new operational capabilities, allowing the agent to effect change through tools like file systems, browsers, and code execution environments.
- In `bojieli/ai-agent-book`, the `policy` method in [`agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/agent.py) demonstrates the practical interface: consuming `Observation` objects and emitting `ActionModel` instances in a continuous loop.

## Frequently Asked Questions

### Can an LLM agent work with a limited observation space?

Yes, but performance degrades significantly. According to the source analysis, if the observation space is too narrow, the model may lack essential data and fail to reason correctly, even when it possesses the necessary internal knowledge. The observation space acts as the agent's window into the environment—restricting it is akin to blindfolding the system.

### How do I decide whether to expand the observation space or action space?

Consult the troubleshooting guidance in [`book-en/reference-answers.md`](https://github.com/bojieli/ai-agent-book/blob/main/book-en/reference-answers.md). If the agent demonstrates ignorance of available information or asks questions that the environment could answer, expand the **observation space**. If the agent clearly understands what steps to take but cannot execute them (responding only with natural language descriptions), expand the **action space** by adding tools or commands.

### What happens if the action space is too broad?

An excessively broad action space can lead to safety issues and operational errors. While the book emphasizes that a limited action space forces the model into natural language suggestions, an unrestricted space might allow destructive operations. The design should balance capability with appropriate guardrails and validation layers.

### Are observation space and action space fixed after deployment?

No. As illustrated by the Manus example in Chapter 1, production systems often dynamically enlarge both spaces. Adding a virtual browser expands the observation space; adding file system access expands the action space. These modifications allow existing models to handle new tasks without retraining, making space expansion a key systems-engineering strategy for capability enhancement.