# How the Open Notebook ContextBuilder Decides Which Context to Send to AI Models

> Discover how the Open Notebook ContextBuilder selects optimal context for AI models using its priority-weighted pipeline configurable rules deduplication and token-budget truncation.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-06-24

---

**The ContextBuilder uses a priority-weighted pipeline that filters sources, notes, and insights through configurable inclusion rules, deduplication, and token-budget truncation to assemble the optimal context window for AI models.**

The `ContextBuilder` class in the `lfnovo/open-notebook` repository serves as the central intelligence for assembling context payloads sent to large language models. Understanding how ContextBuilder decides which context to send to AI models requires examining its multi-stage pipeline that balances informational completeness against strict token constraints.

## Parameter Intake and Configuration Defaults

When instantiated, the builder captures all decision parameters through its `__init__` method in [`open_notebook/utils/context_builder.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/context_builder.py) (lines 65-78). These include `source_id`, `notebook_id`, `include_insights`, `include_notes`, `context_config`, and `max_tokens`.

If no `ContextConfig` object is supplied, the builder initializes default settings at lines 88-99:

- **Insights**: Enabled by default (`True`)
- **Notes**: Enabled by default (`True`)
- **Priority weights**: Source (100), Note (50), Insight (75)

These defaults establish the baseline for what content categories receive preferential treatment in the final context assembly.

## The Build Pipeline Orchestration

The `build()` method (lines 105-135) orchestrates the entire decision sequence:

1. Clears previous context items
2. Adds source-specific context (if `source_id` provided)
3. Adds notebook-specific context (if `notebook_id` provided)
4. Processes custom parameters
5. Removes duplicates
6. Prioritizes remaining items
7. Truncates to token budget

This pipeline ensures deterministic context assembly regardless of input complexity.

## Selecting Source Context for AI Models

When a `source_id` is present, `_add_source_context()` (lines 45-70) determines inclusion through **inclusion levels**:

- `"insights"`: Include only extracted insights
- `"full content"`: Include complete source text
- `"not in"`: Exclude entirely

The `context_size` parameter further refines selection:

- `"short"`: Uses summarized content
- `"long"`: Uses full text content

When `include_insights` is `True`, each insight attached to the source becomes its own `ContextItem` (lines 82-100), allowing granular control over analytical content.

## Assembling Notebook Context

For notebook-scoped queries, `_add_notebook_context()` implements sophisticated filtering:

**Source Selection via ContextConfig**

The `ContextConfig.sources` map (lines 22-28) enables per-source inclusion control. If no map exists, the builder defaults to including all notebook sources with insights-only level.

**Note Integration**

When `include_notes` is enabled, the builder consults `ContextConfig.notes` (lines 34-44). Each selected note passes through `_add_note_context()` (lines 54-80), which mirrors source logic: validating IDs, fetching via `Note.get`, and selecting short or full content based on inclusion specifications.

## Custom Parameter Processing

Any kwargs prefixed with `custom_` are logged for future extensions via `_process_custom_params()` (lines 96-104). This extensibility hook allows the pipeline to accommodate additional filtering logic without modifying core builder methods.

## Optimization and Token Budget Management

After content collection, three optimization layers ensure efficient AI context windows:

**Deduplication**

The `remove_duplicates()` method (lines 151-162) eliminates items sharing identical IDs, preventing redundant transmission of sources, notes, or insights that appear multiple times in the assembly chain.

**Priority Weighting**

The `prioritize()` method (lines 164-169) sorts items by their `priority` field using weights from `ContextConfig.priority_weights`. Higher values appear earlier in the context, ensuring critical information survives truncation.

**Token Budget Enforcement**

When `max_tokens` is specified, `truncate_to_fit()` (lines 171-196) implements the final filter. It calculates cumulative token counts using the `token_count` field established at item creation, then removes lowest-priority items first until the total fits within budget. This guarantees the AI model receives only what it can process without cutoff mid-document.

## Response Formatting

The `_format_response()` method (lines 198-224) structures the final payload by grouping items into sources, notes, and insights categories. It attaches metadata including original config flags and inclusion totals, providing transparency into what the ContextBuilder decided to send to the AI models.

## Implementation Code Examples

```python

# Build context for a notebook, limiting to 2,000 tokens

await build_notebook_context(
    notebook_id="notebook:12345",
    max_tokens=2000,
)

# Build context for a single source with insights disabled

await build_source_context(
    source_id="source:abcde",
    include_insights=False,
)

# Mixed context with custom priority weighting

custom_cfg = ContextConfig(
    sources={"source:xyz": "full content"},
    notes={"note:789": "full content"},
    priority_weights={"source": 120, "note": 80, "insight": 60},
    max_tokens=1500,
)
await ContextBuilder(
    notebook_id="notebook:12345",
    context_config=custom_cfg,
).build()

```

## Summary

The ContextBuilder decides which context to send to AI models through a systematic pipeline:

- **Explicit parameters** (`source_id`, `notebook_id`) guarantee specific entity inclusion
- **Configuration maps** (`ContextConfig.sources`, `ContextConfig.notes`) define per-item inclusion levels
- **Category flags** (`include_insights`, `include_notes`) enable/disable entire content types
- **Priority weights** determine ordering for token budget survival
- **Deduplication** prevents redundant content transmission
- **Token truncation** enforces hard limits by removing low-priority items first

This architecture ensures optimal context utilization while respecting AI model constraints.

## Frequently Asked Questions

### How does ContextBuilder prioritize different content types?

The ContextBuilder assigns numeric priority values to each `ContextItem` based on `ContextConfig.priority_weights`. By default, sources receive weight 100, insights 75, and notes 50. Higher values ensure content appears earlier in the context window, surviving truncation when token limits are enforced. These weights are applied during the `prioritize()` method (lines 164-169) before final assembly.

### What happens when the context exceeds the max_tokens limit?

When a `max_tokens` budget is specified, the `truncate_to_fit()` method (lines 171-196) removes items starting from the lowest priority end of the list. It calculates cumulative token counts using the `token_count` field established at item creation, continuing removal until the total fits within budget. This ensures high-priority sources and insights remain in the final payload sent to AI models.

### How does ContextBuilder handle duplicate content across sources and notes?

The `remove_duplicates()` method (lines 151-162) filters the context list by item ID, ensuring that identical sources, notes, or insights appearing through multiple inclusion paths (such as both explicit `source_id` and notebook membership) are transmitted only once. This prevents token waste and model confusion from redundant content.

### Can I customize which sources are included without modifying the code?

Yes, through the `ContextConfig.sources` dictionary. When instantiating `ContextBuilder`, pass a `ContextConfig` object with a `sources` map specifying each source ID and its desired inclusion level (`"insights"`, `"full content"`, or `"not in"`). If no map is provided, the builder defaults to including all notebook sources with insights-only level, as implemented in `_add_notebook_context()` (lines 22-28).