AI Context Selection with Sources and Notebooks in Open Notebook

Open Notebook uses the ContextBuilder utility to assemble token-aware, prioritized payloads from selected sources, notes, and insights, ensuring language models receive only the most relevant research context while respecting budget constraints.

Open Notebook is an open-source research platform that gives you precise control over what information reaches your AI models. The AI context selection system lets you curate exactly which sources, notes, and insights get sent to the language model, preventing token waste and improving response quality. At the heart of this system lies the ContextBuilder class in open_notebook/utils/context_builder.py, which orchestrates the retrieval, prioritization, and formatting of research data.

How ContextBuilder Assembles AI Context

The context selection pipeline follows a structured eight-step process inside open_notebook/utils/context_builder.py. Each step handles specific data transformations to build a clean, deduplicated payload ready for LLM consumption.

Initialization and Configuration

The process begins when you instantiate ContextBuilder with flexible keyword arguments including source_id, notebook_id, include_insights, and max_tokens. In the __init__ method (lines 65‑87), these arguments are stored in self.params and extracted into attributes for later use.

from open_notebook.utils.context_builder import ContextBuilder, ContextConfig

# Configure with specific requirements

config = ContextConfig(
    include_insights=True,
    include_notes=True,
    max_tokens=2000
)
builder = ContextBuilder(
    notebook_id="nb:123",
    context_config=config,
    max_tokens=2000
)

Building the Context Pipeline

Calling await builder.build() (lines 105‑135) initiates the assembly process. This method clears previous items and delegates to three private helper methods: _add_source_context, _add_notebook_context, and _process_custom_params.

Source Handling and Insight Extraction

For each source ID specified, the builder loads the Source record via Source.get and retrieves either short or long representations using source.get_context. If include_insights is enabled, every SourceInsight is added as a separate ContextItem. This logic resides in _add_source_context (lines 142‑200).

# Inside _add_source_context - pseudocode representation

for source_id in self.params.get('source_ids', []):
    source = await Source.get(source_id)
    context_items.append(source.get_context(size="long"))
    if self.include_insights:
        for insight in source.insights:
            context_items.append(ContextItem(
                content=insight.content,
                type="insight",
                priority=75
            ))

Notebook and Note Processing

When processing notebooks via _add_notebook_context (lines 210‑258), the builder fetches the Notebook record and adds its configured sources and notes. Without explicit configuration, it defaults to "all sources with insights" and "all notes with full content". Individual notes are retrieved via Note.get in _add_note_context (lines 254‑288), where short or long context representations are selected based on the configuration.

Deduplication and Token Truncation

The final processing stage (lines 311‑368) ensuring efficiency involves three critical operations:

• remove_duplicates: Eliminates redundant content items
• prioritize: Sorts items by priority weights defined in ContextConfig.priority_weights (source 100, insight 75, note 50)
• truncate_to_fit: Trims the lowest-priority items until the token budget specified by max_tokens is satisfied

Response Formatting

The _format_response method (lines 374‑416) groups final items by type (sources, notes, insights) and attaches meta-information including total token counts, counts per type, and configuration snapshots.

Convenience Helpers

The module exports three async convenience functions (lines 421‑496) that hide boilerplate: build_notebook_context, build_source_context, and build_mixed_context.

Token Budgeting and Priority Weights

Each ContextItem automatically computes its token count using token_count from open_notebook/utils/token_utils.py. The priority system ensures high-value content survives when budgets tighten. By default, sources receive priority 100, insights receive 75, and notes receive 50. This hierarchy ensures foundational source material reaches the model even when you must drop supplementary notes and insights to fit within max_tokens constraints.

Long-Form Context Generation in Notebooks

Beyond the builder's structured payload, Notebook.get_context in open_notebook/domain/notebook.py (lines 71‑128) constructs a human-readable narrative for UI features like podcast generation. This method concatenates source titles, full text, and insight bullet-lists into a cohesive string:

# open_notebook/domain/notebook.py, lines 71-128

async def get_context(self) -> str:
    sources = await self.get_sources(include_full_text=True)
    notes = await self.get_notes(include_content=True)
    context_blocks = []
    # ... formatting logic ...

    return "\n\n".join(context_blocks)

When ContextBuilder receives context_size="long", it leverages this narrative representation rather than fragmented items.

Practical Implementation Examples

Basic Notebook Context Without Limits

Retrieve full context for a specific notebook when token costs are not a concern:

from open_notebook.utils.context_builder import build_notebook_context

async def get_notebook_context():
    ctx = await build_notebook_context(notebook_id="nb:123")
    # Access structured data

    sources = ctx['sources']
    notes = ctx['notes']
    insights = ctx['insights']
    total_tokens = ctx['meta']['total_tokens']
    return ctx

Token-Limited Context with Selective Inclusion

Limit to 1500 tokens while including insights but excluding notes:

from open_notebook.utils.context_builder import ContextConfig, build_notebook_context

async def get_limited_context():
    cfg = ContextConfig(include_insights=True, include_notes=False)
    ctx = await build_notebook_context(
        notebook_id="nb:123",
        context_config=cfg,
        max_tokens=1500,
    )
    return ctx

Mixed Source and Note Selection

Combine specific sources and notes from different notebooks:

from open_notebook.utils.context_builder import build_mixed_context

async def custom_context():
    ctx = await build_mixed_context(
        source_ids=["src:abc", "src:def"],
        note_ids=["note:xyz"],
        max_tokens=2000,
    )
    return ctx

Advanced Direct Builder Usage

For complex scenarios requiring granular control over individual item representations:

from open_notebook.utils.context_builder import ContextBuilder, ContextConfig

async def manual_builder():
    cfg = ContextConfig(
        sources={"src:abc": "insights", "src:def": "full content"},
        notes={"note:xyz": "full content"},
        max_tokens=2500,
    )
    builder = ContextBuilder(
        notebook_id="nb:123",
        context_config=cfg,
        max_tokens=2500,
    )
    result = await builder.build()
    return result

All examples require an async runtime such as FastAPI or an asyncio event loop.

Key Files in the Context Selection Architecture

File	Role
open_notebook/utils/context_builder.py	Core assembly engine containing ContextBuilder, item dataclasses, and helper functions.
open_notebook/domain/notebook.py	Notebook model with get_context method for long-form narrative generation.
open_notebook/domain/source.py	Source model with insight fetching and context representation methods.
open_notebook/domain/note.py	Note model with content retrieval capabilities.
open_notebook/utils/token_utils.py	Token-counting implementation used by ContextItem.
open_notebook/api/routers/context.py	API router exposing context-building endpoints (if present).
open_notebook/ai/provision.py	LLM provider selection that consumes built context payloads.
open_notebook/graphs/ask.py	LangGraph workflow integrating context building before LLM invocation.

Summary

• The ContextBuilder class in open_notebook/utils/context_builder.py orchestrates AI context selection by fetching, prioritizing, and formatting research data.
• The system processes sources, notes, and insights as distinct primitives with configurable priority weights (100, 50, and 75 respectively).
• Token budgeting occurs through automatic counting and truncation, ensuring payloads respect max_tokens limits while preserving high-priority items.
• Async convenience functions (build_notebook_context, build_source_context, build_mixed_context) simplify common usage patterns.
• Deduplication and sorting happen automatically before final formatting into a structured response with metadata.

Frequently Asked Questions

How does Open Notebook prevent sending duplicate content to the LLM?

The ContextBuilder automatically removes duplicates during the post-processing stage (lines 311‑368) before final formatting. This ensures that if a source appears both individually and as part of a notebook configuration, it only consumes tokens once in the final payload.

Can I mix sources and notes from different notebooks in a single context request?

Yes. The build_mixed_context convenience function (lines 421‑496) accepts explicit source_ids and note_ids arrays, allowing you to combine research materials from multiple notebooks regardless of their original organizational boundaries.

What happens if my selected content exceeds the max_tokens limit?

The builder calls truncate_to_fit (lines 311‑368), which removes lowest-priority items first based on ContextConfig.priority_weights. Sources (priority 100) are preserved longest, followed by insights (75) and notes (50), until the total token count falls within budget.

Where does the token counting logic reside?

Token counting is implemented in open_notebook/utils/token_utils.py and invoked automatically when ContextItem objects are created. This ensures accurate budget enforcement before content reaches the LLM provider configured in open_notebook/ai/provision.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}