# OKF Agent Prompt Templates for BigQuery and Web Ingestion Tasks

> Discover OKF agent prompt templates for BigQuery and web ingestion. Learn how reference_instruction.md and web_ingestion_instruction.md streamline metadata enrichment and web crawling tasks for efficient data handling.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: how-to-guide
- Published: 2026-07-16

---

**The OKF agent relies on two specialized markdown prompt templates—[`reference_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/reference_instruction.md) for BigQuery metadata enrichment and [`web_ingestion_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/web_ingestion_instruction.md) for web crawling workflows—located in the `okf/src/reference_agent/prompts/` directory and dynamically loaded at runtime.**

The GoogleCloudPlatform/knowledge-catalog repository implements an Open Knowledge Format (OKF) agent that automates the creation of structured documentation for data assets. Understanding the specific prompt templates for BigQuery and web ingestion tasks is essential for customizing the agent’s behavior or debugging its output. These templates provide the LLM with precise instructions for metadata extraction, content synthesis, and citation handling.

## BigQuery Enrichment Prompt Template

### Template Location and Purpose

The **BigQuery enrichment workflow** uses the [`reference_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/reference_instruction.md) template stored at [`okf/src/reference_agent/prompts/reference_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/prompts/reference_instruction.md). This file provides the instruction set for the **reference agent** that creates or augments OKF documents for BigQuery assets such as datasets and tables.

The template directs the agent to read existing documentation, fetch raw metadata from BigQuery, optionally sample rows, and emit a single `write_concept_doc` call containing the required front-matter and body sections. It ensures consistent formatting and content structure across all generated BigQuery asset documentation.

### Runtime Loading Mechanism

At runtime, the agent loads this template via the `_load_prompt` utility function defined in the codebase. The function leverages Python’s `importlib.resources` to read the bundled markdown file:

```python
from importlib import resources

def _load_prompt(filename: str) -> str:
    # Resources are bundled with the package; the function loads the markdown file.

    return resources.files("reference_agent.prompts").joinpath(filename).read_text()

```

In the reference agent implementation ([`okf/src/reference_agent/agent.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/agent.py)), the template is loaded as follows:

```python

# In the reference agent (e.g., okf/src/reference_agent/agent.py)

instruction = _load_prompt("reference_instruction.md")

```

The resulting `instruction` string is then interpolated into the LLM request’s system or user messages to guide the agent’s BigQuery-specific behavior.

## Web Ingestion Prompt Template

### Template Location and Purpose

For **web ingestion tasks**, the OKF agent utilizes [`web_ingestion_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/web_ingestion_instruction.md), located at [`okf/src/reference_agent/prompts/web_ingestion_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/prompts/web_ingestion_instruction.md). This template defines the workflow for the **web-ingestion agent** responsible for crawling seed URLs and extracting authoritative reference concepts.

The template specifies budgeting constraints, link-filtering criteria, augmentation rules, and required citation handling. It enables the agent to either enrich existing concepts or mint new reference concepts—such as metrics, dimensions, and joins—derived from web content.

### Integration with the Agent Pipeline

Similar to the BigQuery template, the web ingestion prompt is loaded using the same `_load_prompt` function:

```python
instruction = _load_prompt("web_ingestion_instruction.md")

```

This instruction string configures the LLM to execute the multi-step web crawling and content extraction workflow defined in the template.

## Technical Implementation and Supporting Files

The prompt loading architecture centers on the `_load_prompt` function, which reads files from the `reference_agent.prompts` package namespace. This approach ensures that prompt templates are treated as package resources, maintaining portability across different deployment environments.

Beyond the prompt files themselves, the [`okf/src/reference_agent/agent.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/agent.py) module orchestrates the loading and assembly of these templates into final LLM prompts. Additionally, [`okf/src/reference_agent/bundle/synthesizer.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/bundle/synthesizer.py) utilizes a `_PROMPT_TEMPLATE` constant alongside the loaded instruction files to synthesize final document bundles.

| File | Role |
|------|------|
| [`okf/src/reference_agent/prompts/reference_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/prompts/reference_instruction.md) | Prompt template for BigQuery metadata-driven enrichment. |
| [`okf/src/reference_agent/prompts/web_ingestion_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/prompts/web_ingestion_instruction.md) | Prompt template for crawling and ingesting web pages. |
| [`okf/src/reference_agent/agent.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/agent.py) | Loads templates and assembles the final prompt sent to the model. |
| [`okf/src/reference_agent/bundle/synthesizer.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/bundle/synthesizer.py) | Uses `_PROMPT_TEMPLATE` together with loaded instruction files for bundle synthesis. |

## Summary

- The OKF agent employs two distinct prompt templates: [`reference_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/reference_instruction.md) for BigQuery tasks and [`web_ingestion_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/web_ingestion_instruction.md) for web ingestion workflows.
- Both templates reside in `okf/src/reference_agent/prompts/` and are loaded at runtime via the `_load_prompt` function using `importlib.resources`.
- The [`reference_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/reference_instruction.md) template guides the creation of structured documentation for BigQuery datasets and tables, including metadata extraction and row sampling.
- The [`web_ingestion_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/web_ingestion_instruction.md) template governs web crawling logic, link filtering, and citation handling for deriving reference concepts from URLs.
- Core implementation files include [`okf/src/reference_agent/agent.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/agent.py) for orchestration and [`okf/src/reference_agent/bundle/synthesizer.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/bundle/synthesizer.py) for document synthesis.

## Frequently Asked Questions

### What is the difference between the BigQuery and web ingestion prompt templates?

The **BigQuery template** ([`reference_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/reference_instruction.md)) focuses on metadata-driven enrichment of database assets, instructing the agent to fetch schema metadata and sample rows. The **web ingestion template** ([`web_ingestion_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/web_ingestion_instruction.md)) handles URL crawling, content extraction, and citation management for deriving knowledge from web sources.

### Where are the OKF agent prompt templates stored in the repository?

Both templates are located under `okf/src/reference_agent/prompts/` within the GoogleCloudPlatform/knowledge-catalog repository. Specifically, [`reference_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/reference_instruction.md) handles BigQuery workflows, while [`web_ingestion_instruction.md`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/web_ingestion_instruction.md) manages web ingestion tasks.

### How does the OKF agent load prompt templates at runtime?

The agent calls the `_load_prompt` function, which uses Python’s `importlib.resources` to read markdown files from the `reference_agent.prompts` package. This method ensures reliable access to template files regardless of the execution environment.

### Which source files handle the prompt template loading for the OKF agent?

The primary loading logic resides in [`okf/src/reference_agent/agent.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/agent.py), which invokes `_load_prompt` to retrieve template content. The [`okf/src/reference_agent/bundle/synthesizer.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/bundle/synthesizer.py) file additionally uses these templates alongside internal constants like `_PROMPT_TEMPLATE` to generate final output documents.