Hindsight Memory Types Explained: World, Experience, and Opinion

Hindsight organizes all stored information into three primary memory types—world (objective facts), experience (personal events), and opinion/observation (preferences with confidence scores)—to enable AI agents to retrieve contextually appropriate knowledge.

The open-source Hindsight memory system (vectorize-io/hindsight) uses these categorical distinctions to determine how facts are stored, weighted, and recalled. Each Hindsight memory type serves a distinct purpose in building an agent's understanding of static knowledge, personal history, and subjective user preferences.

Understanding the Three Hindsight Memory Types

Hindsight classifies every retained fact using a strict schema enforced at the database level. In hindsight-api-slim/hindsight_api/models.py, a CheckConstraint on line 123 validates that all entries use one of the allowed literals defined in the codebase.

World Memory: Objective, Verifiable Facts

The world memory type stores static, externally verifiable information that exists independently of user interactions. This includes encyclopedic knowledge, public data, definitions, and background context about people, places, or things.

According to the FactExtraction model in hindsight-api-slim/hindsight_api/engine/retain/fact_extraction.py (line 90), the world literal represents facts like "Alice works at Google" or "Paris is the capital of France." These entries lack timestamps because they describe persistent states rather than events.

Experience Memory: Personal, Time-Bound Events

The experience type captures subjective, temporal events that the user or agent has personally observed or participated in. This includes conversation history, actions taken, tasks performed, and any narrative reflecting what happened from the user's point of view.

As documented in README.md (line 191), experience memories build a personal timeline that agents reference for continuity. Examples include "Yesterday I ran 5 km" or "I fixed a bug in the code this morning." The retain pipeline automatically extracts the experience label from LLM output when processing personal narratives.

Opinion and Observation Memory: Preferences with Confidence

The opinion type (deprecated in favor of observation) stores subjective preferences, beliefs, and inferred sentiments alongside a confidence score. These memories guide decision-making and reflection by capturing what the system has learned about user tastes.

In hindsight-api-slim/hindsight_api/models.py (lines 126-128), the schema implements a partial index requiring confidence_score for all opinion-type entries. The newer observation type is a superset that internally maps to the same storage but better reflects that these are agent-formed observations about user preferences.

When to Use Each Hindsight Memory Type

Choosing the correct memory type determines how the agent retrieves and weights information during reasoning tasks.

Use world when storing static, externally verifiable knowledge that does not depend on who is asking. This includes factual background, organizational hierarchies, or domain definitions that remain true regardless of the user.

Use experience for dynamic, personal data including interaction logs, action histories, and timestamped narratives. This type powers personal timelines and context-aware continuity in conversations.

Use opinion or observation for preference modeling and sentiment tracking. Because these entries carry confidence scores (e.g., "User prefers tea over coffee — confidence 0.9"), they are ideal for weighted decision-making during agent reflection.

Working with Memory Types in Code

The Hindsight Python client and low-level MemoryEngine API allow explicit control over memory classification during retention and filtering during recall.

Retaining Facts with Specific Types

The retain method automatically infers the appropriate type from content, but you can specify it explicitly via the fact_type parameter:

from hindsight_client import HindsightClient

client = HindsightClient()

# World fact - automatically tagged or explicitly set

client.retain(
    bank_id="my-bank",
    content="The Eiffel Tower is 324 meters tall.",
    fact_type="world"  # Optional: inferred from content

)

# Experience - personal event with implicit timestamp

client.retain(
    bank_id="my-bank",
    content="I ran 5 km this morning.",
    fact_type="experience"
)

# Opinion/Observation - preference with confidence

client.retain(
    bank_id="my-bank",
    content="I prefer tea over coffee.",
    fact_type="opinion"  # Maps internally to observation

)

The retain endpoint definition in hindsight-clients/python/hindsight_client_api/api/memory_api.py (line 56) documents the optional type parameter used for explicit classification.

Filtering Recall by Memory Type

The recall API accepts a types parameter to retrieve only relevant memory slices for specific tasks:

# Retrieve only objective background knowledge

world_facts = client.recall(bank_id="my-bank", types=["world"])

# Retrieve personal history for continuity

experiences = client.recall(bank_id="my-bank", types=["experience"])

# Retrieve preferences for decision-making

preferences = client.recall(bank_id="my-bank", types=["observation"])

This filtering logic is implemented in hindsight-api-slim/hindsight_api/engine/memory_engine.py at line 2260, where the engine applies type constraints during the recall pipeline.

Direct MemoryEngine API Usage

For low-level control, the MemoryEngine class exposes the same type system:

from hindsight_api import MemoryEngine

engine = MemoryEngine(database_url="postgresql://...")

# Explicit type insertion bypasses LLM inference

engine.retain(
    bank_id="my-bank",
    content="Alice joined the engineering team in 2023.",
    fact_type="world"
)

engine.retain(
    bank_id="my-bank",
    content="Alice deployed the new feature yesterday.",
    fact_type="experience"
)

Summary

• World memory stores objective, verifiable facts independent of user identity, enforced by the database schema in models.py.
• Experience memory captures timestamped, personal events and interaction histories for timeline construction.
• Opinion/Observation memory holds subjective preferences with mandatory confidence scores, indexed separately for weighted retrieval.
• The Hindsight API supports explicit type declaration during retention and type filtering during recall, as documented in memory_api.py and implemented in memory_engine.py.

Frequently Asked Questions

What is the difference between opinion and observation in Hindsight?

Observation is the newer, preferred name for the opinion memory type. While opinion remains in the codebase for backward compatibility, the observation type better represents that these are agent-formed observations about user preferences. Both map to the same underlying storage and require a confidence_score field as defined in models.py lines 126-128.

Can I filter recall requests to exclude specific Hindsight memory types?

Yes, the recall API supports type filtering but does not support exclusion lists directly. You must specify the positive types you want to include using the types parameter (e.g., types=["world", "experience"] to exclude opinions). To exclude a type, request all other allowed types explicitly.

How does Hindsight automatically classify content into memory types?

The retain pipeline uses an LLM-based extraction model to infer the fact type from content semantics. The FactExtraction model in fact_extraction.py (line 90) defines the allowed literals (world, experience, opinion), and the system tags incoming text accordingly before the database CheckConstraint in models.py (line 123) validates the classification.

What happens if I specify an invalid memory type in the API?

The request fails with a validation error before reaching the database. The Pydantic model in fact_extraction.py validates the fact_type literal against the allowed set, and the database schema enforces the constraint at the storage layer. Only world, experience, opinion, and observation are accepted values.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}