# list_concepts vs read_concept_raw: Functional Differences in the OKF Reference Agent

> Discover the functional difference between list_concepts and read_concept_raw in the OKF reference agent. Learn how to enumerate concepts or retrieve specific metadata effectively.

- Repository: [Google Cloud Platform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog)
- Tags: deep-dive
- Published: 2026-07-16

---

**The functional difference between list_concepts and read_concept_raw is that list_concepts enumerates all available concepts from the active source for discovery purposes, while read_concept_raw retrieves detailed metadata for a single specific concept identified by its concept_id.**

The **OKF reference agent** in the [GoogleCloudPlatform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog) repository exposes these two tools to LLMs for catalog interaction. Understanding when to use **list_concepts** versus **read_concept_raw** is essential for building effective data discovery workflows against BigQuery tables, views, and other data assets.

## Core Functional Differences

Both tools operate on the active source configured in the agent's context, but serve distinct purposes in the data discovery lifecycle:

- **list_concepts**: Performs **catalog enumeration** by querying the active source for all concepts it knows about. It returns a list of lightweight dictionaries containing `id`, `type`, `resource`, and `hint` fields for each asset. Use this when you need to **discover** what tables, views, or datasets are available.

- **read_concept_raw**: Performs **metadata retrieval** for a single concept identified by its slash-joined `concept_id`. It returns a dictionary containing source-specific structured metadata such as schema definitions, partitioning configuration, clustering keys, row counts, and timestamps. Use this when you already know the exact asset you want to inspect.

## Implementation Details in source_tools.py

The concrete implementations reside in [`okf/src/reference_agent/tools/source_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/tools/source_tools.py), where each tool follows a distinct execution path through the source abstraction layer.

### How list_concepts Enumerates Assets

The `list_concepts()` function obtains the active source from the agent's context via `get_context().source` and delegates to the source's enumeration method:

1. Calls `src.list_concepts()` to retrieve `ConceptRef` objects representing all available assets
2. Transforms each reference into a plain dictionary using the `_ref_to_dict` helper
3. Returns a list of dictionaries optimized for discovery workflows

The output contains summary fields only—enough to identify assets without the overhead of full metadata extraction.

### How read_concept_raw Retrieves Detailed Metadata

The `read_concept_raw(concept_id)` function performs targeted metadata extraction through a three-step resolution process:

1. Parses the slash-joined `concept_id` using `parse_concept_id` to reconstruct the concept identifier
2. Locates the corresponding `ConceptRef` via `src.find(cid)` 
3. Fetches raw metadata by calling `src.read_concept(ref)`

If the concept cannot be found, the function raises a `ValueError`. The returned dictionary contains the complete source-specific metadata payload, which for BigQuery tables includes schema fields, partitioning specifications, clustering columns, and storage statistics.

## Working with Concept Identifiers

The tools share a dependency on **slash-joined concept identifiers** (e.g., `"tables/events_"`). The `list_concepts` tool returns these identifiers in the `id` field of each dictionary, and `read_concept_raw` consumes them via its `concept_id` parameter.

The internal `parse_concept_id` function handles the parsing logic, while `src.find(cid)` resolves the identifier back to a `ConceptRef` object before metadata retrieval occurs.

## Practical Code Examples

### Example 1: Discover Available Assets

```python

# Enumerate all concepts from the active source

concepts = list_concepts()
for concept in concepts:
    print(f"{concept['id']} – {concept['type']}")

```

### Example 2: Retrieve Specific Metadata

```python

# Get raw metadata for a known concept

concept_id = "tables/events_"  # Must be an id returned by list_concepts

metadata = read_concept_raw(concept_id)

print("Schema:", metadata["schema"])
print("Partitioning:", metadata.get("partitioning"))
print("Row count:", metadata.get("rowCount"))

```

### Example 3: Chaining Discovery and Retrieval

```python

# Workflow: enumerate all tables and inspect schema complexity

all_concepts = list_concepts()
table_ids = [c["id"] for c in all_concepts if c["type"] == "BigQuery Table"]

for cid in table_ids:
    try:
        md = read_concept_raw(cid)
        field_count = len(md.get("schema", []))
        print(f"{cid}: {field_count} fields")
    except ValueError:
        print(f"Concept {cid} not found")

```

## Source Code Architecture

Understanding the functional split requires examining these key files in the repository:

- **[`okf/src/reference_agent/tools/source_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/tools/source_tools.py)**: Implements both `list_concepts` and `read_concept_raw` with their respective helper functions `_ref_to_dict` and `parse_concept_id`.

- **[`okf/src/reference_agent/agent.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/agent.py)**: Registers both tools with the LLM via `FunctionTool` wrappers, making them available for agent invocation.

- **[`okf/src/reference_agent/sources/base.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/base.py)**: Defines the abstract interface requiring concrete sources to implement `list_concepts()` and `read_concept(ref)` methods.

- **[`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py)**: Provides the concrete BigQuery implementation that returns table and view references during enumeration and extracts full table metadata during retrieval.

## Summary

- **list_concepts** is an **enumeration operation** designed for catalog browsing and discovery, returning lightweight summaries of all available assets.

- **read_concept_raw** is a **detail lookup operation** that requires a specific `concept_id` and returns complete structured metadata for that single asset.

- Both tools access the **active source** configured in the agent context (`get_context().source`) and delegate to source-specific implementations.

- The tools are designed to be used sequentially: use `list_concepts` to discover asset identifiers, then `read_concept_raw` to inspect specific assets of interest.

- Error handling differs significantly: `list_concepts` returns empty lists when no assets exist, while `read_concept_raw` raises `ValueError` for invalid concept identifiers.

## Frequently Asked Questions

### Can I call read_concept_raw without using list_concepts first?

Yes, provided you know the valid `concept_id` string. The identifier follows a slash-joined format (e.g., `"tables/dataset_name_table_name"`) that the source can resolve. However, since these identifiers are source-specific, using `list_concepts` first is the recommended approach to ensure valid identifiers.

### What specific metadata fields does read_concept_raw return for BigQuery tables?

According to the BigQuery source implementation in [`okf/src/reference_agent/sources/bigquery.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/sources/bigquery.py), the returned dictionary includes `schema` (list of field definitions), `partitioning` (time or range partitioning config), `clustering` (clustering column names), `rowCount`, `creationTime`, `lastModifiedTime`, and other storage statistics. The exact fields depend on the source type.

### Why does list_concepts return dictionaries instead of raw ConceptRef objects?

The `_ref_to_dict` helper in [`source_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/source_tools.py) serializes `ConceptRef` objects into plain dictionaries to ensure compatibility with LLM function calling interfaces. This transformation strips internal Python objects while preserving the essential `id`, `type`, `resource`, and `hint` fields needed for asset identification.

### Where is the error handling implemented when a concept is not found?

The `read_concept_raw` function in [`okf/src/reference_agent/tools/source_tools.py`](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/src/reference_agent/tools/source_tools.py) explicitly checks if `src.find(cid)` returns a valid reference. If the lookup returns `None` or fails to locate the concept, the function raises a `ValueError` with a descriptive message indicating that the concept_id could not be found in the active source.