# AI Context Selection with Sources and Notebooks in Open Notebook

> Enhance AI context selection in Open Notebook using ContextBuilder. Prioritize relevant research from sources and notebooks while managing budget for efficient language model use.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-07-02

---

**Open Notebook uses the `ContextBuilder` utility to assemble token-aware, prioritized payloads from selected sources, notes, and insights, ensuring language models receive only the most relevant research context while respecting budget constraints.**

Open Notebook is an open-source research platform that gives you precise control over what information reaches your AI models. The **AI context selection** system lets you curate exactly which sources, notes, and insights get sent to the language model, preventing token waste and improving response quality. At the heart of this system lies the `ContextBuilder` class in [`open_notebook/utils/context_builder.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/context_builder.py), which orchestrates the retrieval, prioritization, and formatting of research data.

## How ContextBuilder Assembles AI Context

The context selection pipeline follows a structured eight-step process inside [`open_notebook/utils/context_builder.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/context_builder.py). Each step handles specific data transformations to build a clean, deduplicated payload ready for LLM consumption.

### Initialization and Configuration

The process begins when you instantiate `ContextBuilder` with flexible keyword arguments including `source_id`, `notebook_id`, `include_insights`, and `max_tokens`. In the `__init__` method (lines 65‑87), these arguments are stored in `self.params` and extracted into attributes for later use.

```python
from open_notebook.utils.context_builder import ContextBuilder, ContextConfig

# Configure with specific requirements

config = ContextConfig(
    include_insights=True,
    include_notes=True,
    max_tokens=2000
)
builder = ContextBuilder(
    notebook_id="nb:123",
    context_config=config,
    max_tokens=2000
)

```

### Building the Context Pipeline

Calling `await builder.build()` (lines 105‑135) initiates the assembly process. This method clears previous items and delegates to three private helper methods: `_add_source_context`, `_add_notebook_context`, and `_process_custom_params`.

### Source Handling and Insight Extraction

For each source ID specified, the builder loads the `Source` record via `Source.get` and retrieves either short or long representations using `source.get_context`. If `include_insights` is enabled, every `SourceInsight` is added as a separate `ContextItem`. This logic resides in `_add_source_context` (lines 142‑200).

```python

# Inside _add_source_context - pseudocode representation

for source_id in self.params.get('source_ids', []):
    source = await Source.get(source_id)
    context_items.append(source.get_context(size="long"))
    if self.include_insights:
        for insight in source.insights:
            context_items.append(ContextItem(
                content=insight.content,
                type="insight",
                priority=75
            ))

```

### Notebook and Note Processing

When processing notebooks via `_add_notebook_context` (lines 210‑258), the builder fetches the `Notebook` record and adds its configured sources and notes. Without explicit configuration, it defaults to "all sources with insights" and "all notes with full content". Individual notes are retrieved via `Note.get` in `_add_note_context` (lines 254‑288), where short or long context representations are selected based on the configuration.

### Deduplication and Token Truncation

The final processing stage (lines 311‑368) ensuring efficiency involves three critical operations:

- **`remove_duplicates`**: Eliminates redundant content items
- **`prioritize`**: Sorts items by priority weights defined in `ContextConfig.priority_weights` (source 100, insight 75, note 50)
- **`truncate_to_fit`**: Trims the lowest-priority items until the token budget specified by `max_tokens` is satisfied

### Response Formatting

The `_format_response` method (lines 374‑416) groups final items by type (`sources`, `notes`, `insights`) and attaches meta-information including total token counts, counts per type, and configuration snapshots.

### Convenience Helpers

The module exports three async convenience functions (lines 421‑496) that hide boilerplate: `build_notebook_context`, `build_source_context`, and `build_mixed_context`.

## Token Budgeting and Priority Weights

Each `ContextItem` automatically computes its token count using `token_count` from [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py). The priority system ensures high-value content survives when budgets tighten. By default, sources receive priority 100, insights receive 75, and notes receive 50. This hierarchy ensures foundational source material reaches the model even when you must drop supplementary notes and insights to fit within `max_tokens` constraints.

## Long-Form Context Generation in Notebooks

Beyond the builder's structured payload, `Notebook.get_context` in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (lines 71‑128) constructs a human-readable narrative for UI features like podcast generation. This method concatenates source titles, full text, and insight bullet-lists into a cohesive string:

```python

# open_notebook/domain/notebook.py, lines 71-128

async def get_context(self) -> str:
    sources = await self.get_sources(include_full_text=True)
    notes = await self.get_notes(include_content=True)
    context_blocks = []
    # ... formatting logic ...

    return "\n\n".join(context_blocks)

```

When `ContextBuilder` receives `context_size="long"`, it leverages this narrative representation rather than fragmented items.

## Practical Implementation Examples

### Basic Notebook Context Without Limits

Retrieve full context for a specific notebook when token costs are not a concern:

```python
from open_notebook.utils.context_builder import build_notebook_context

async def get_notebook_context():
    ctx = await build_notebook_context(notebook_id="nb:123")
    # Access structured data

    sources = ctx['sources']
    notes = ctx['notes']
    insights = ctx['insights']
    total_tokens = ctx['meta']['total_tokens']
    return ctx

```

### Token-Limited Context with Selective Inclusion

Limit to 1500 tokens while including insights but excluding notes:

```python
from open_notebook.utils.context_builder import ContextConfig, build_notebook_context

async def get_limited_context():
    cfg = ContextConfig(include_insights=True, include_notes=False)
    ctx = await build_notebook_context(
        notebook_id="nb:123",
        context_config=cfg,
        max_tokens=1500,
    )
    return ctx

```

### Mixed Source and Note Selection

Combine specific sources and notes from different notebooks:

```python
from open_notebook.utils.context_builder import build_mixed_context

async def custom_context():
    ctx = await build_mixed_context(
        source_ids=["src:abc", "src:def"],
        note_ids=["note:xyz"],
        max_tokens=2000,
    )
    return ctx

```

### Advanced Direct Builder Usage

For complex scenarios requiring granular control over individual item representations:

```python
from open_notebook.utils.context_builder import ContextBuilder, ContextConfig

async def manual_builder():
    cfg = ContextConfig(
        sources={"src:abc": "insights", "src:def": "full content"},
        notes={"note:xyz": "full content"},
        max_tokens=2500,
    )
    builder = ContextBuilder(
        notebook_id="nb:123",
        context_config=cfg,
        max_tokens=2500,
    )
    result = await builder.build()
    return result

```

All examples require an async runtime such as FastAPI or an asyncio event loop.

## Key Files in the Context Selection Architecture

| File | Role |
|------|------|
| [`open_notebook/utils/context_builder.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/context_builder.py) | Core assembly engine containing `ContextBuilder`, item dataclasses, and helper functions. |
| [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) | Notebook model with `get_context` method for long-form narrative generation. |
| [`open_notebook/domain/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/source.py) | Source model with insight fetching and context representation methods. |
| [`open_notebook/domain/note.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/note.py) | Note model with content retrieval capabilities. |
| [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py) | Token-counting implementation used by `ContextItem`. |
| [`open_notebook/api/routers/context.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/api/routers/context.py) | API router exposing context-building endpoints (if present). |
| [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) | LLM provider selection that consumes built context payloads. |
| [`open_notebook/graphs/ask.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/ask.py) | LangGraph workflow integrating context building before LLM invocation. |

## Summary

- The **`ContextBuilder`** class in [`open_notebook/utils/context_builder.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/context_builder.py) orchestrates **AI context selection** by fetching, prioritizing, and formatting research data.
- The system processes **sources**, **notes**, and **insights** as distinct primitives with configurable priority weights (100, 50, and 75 respectively).
- **Token budgeting** occurs through automatic counting and truncation, ensuring payloads respect `max_tokens` limits while preserving high-priority items.
- **Async convenience functions** (`build_notebook_context`, `build_source_context`, `build_mixed_context`) simplify common usage patterns.
- **Deduplication and sorting** happen automatically before final formatting into a structured response with metadata.

## Frequently Asked Questions

### How does Open Notebook prevent sending duplicate content to the LLM?

The `ContextBuilder` automatically removes duplicates during the post-processing stage (lines 311‑368) before final formatting. This ensures that if a source appears both individually and as part of a notebook configuration, it only consumes tokens once in the final payload.

### Can I mix sources and notes from different notebooks in a single context request?

Yes. The **`build_mixed_context`** convenience function (lines 421‑496) accepts explicit `source_ids` and `note_ids` arrays, allowing you to combine research materials from multiple notebooks regardless of their original organizational boundaries.

### What happens if my selected content exceeds the max_tokens limit?

The builder calls **`truncate_to_fit`** (lines 311‑368), which removes lowest-priority items first based on `ContextConfig.priority_weights`. Sources (priority 100) are preserved longest, followed by insights (75) and notes (50), until the total token count falls within budget.

### Where does the token counting logic reside?

Token counting is implemented in [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py) and invoked automatically when `ContextItem` objects are created. This ensures accurate budget enforcement before content reaches the LLM provider configured in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py).