list_concepts vs read_concept_raw: Functional Differences in the OKF Reference Agent

The functional difference between list_concepts and read_concept_raw is that list_concepts enumerates all available concepts from the active source for discovery purposes, while read_concept_raw retrieves detailed metadata for a single specific concept identified by its concept_id.

The OKF reference agent in the GoogleCloudPlatform/knowledge-catalog repository exposes these two tools to LLMs for catalog interaction. Understanding when to use list_concepts versus read_concept_raw is essential for building effective data discovery workflows against BigQuery tables, views, and other data assets.

Core Functional Differences

Both tools operate on the active source configured in the agent's context, but serve distinct purposes in the data discovery lifecycle:

• list_concepts: Performs catalog enumeration by querying the active source for all concepts it knows about. It returns a list of lightweight dictionaries containing id, type, resource, and hint fields for each asset. Use this when you need to discover what tables, views, or datasets are available.

• read_concept_raw: Performs metadata retrieval for a single concept identified by its slash-joined concept_id. It returns a dictionary containing source-specific structured metadata such as schema definitions, partitioning configuration, clustering keys, row counts, and timestamps. Use this when you already know the exact asset you want to inspect.

Implementation Details in source_tools.py

The concrete implementations reside in okf/src/reference_agent/tools/source_tools.py, where each tool follows a distinct execution path through the source abstraction layer.

How list_concepts Enumerates Assets

The list_concepts() function obtains the active source from the agent's context via get_context().source and delegates to the source's enumeration method:

1. Calls src.list_concepts() to retrieve ConceptRef objects representing all available assets
2. Transforms each reference into a plain dictionary using the _ref_to_dict helper
3. Returns a list of dictionaries optimized for discovery workflows

The output contains summary fields only—enough to identify assets without the overhead of full metadata extraction.

How read_concept_raw Retrieves Detailed Metadata

The read_concept_raw(concept_id) function performs targeted metadata extraction through a three-step resolution process:

1. Parses the slash-joined concept_id using parse_concept_id to reconstruct the concept identifier
2. Locates the corresponding ConceptRef via src.find(cid)
3. Fetches raw metadata by calling src.read_concept(ref)

If the concept cannot be found, the function raises a ValueError. The returned dictionary contains the complete source-specific metadata payload, which for BigQuery tables includes schema fields, partitioning specifications, clustering columns, and storage statistics.

Working with Concept Identifiers

The tools share a dependency on slash-joined concept identifiers (e.g., "tables/events_"). The list_concepts tool returns these identifiers in the id field of each dictionary, and read_concept_raw consumes them via its concept_id parameter.

The internal parse_concept_id function handles the parsing logic, while src.find(cid) resolves the identifier back to a ConceptRef object before metadata retrieval occurs.

Practical Code Examples

Example 1: Discover Available Assets

# Enumerate all concepts from the active source

concepts = list_concepts()
for concept in concepts:
    print(f"{concept['id']} – {concept['type']}")

Example 2: Retrieve Specific Metadata

# Get raw metadata for a known concept

concept_id = "tables/events_"  # Must be an id returned by list_concepts

metadata = read_concept_raw(concept_id)

print("Schema:", metadata["schema"])
print("Partitioning:", metadata.get("partitioning"))
print("Row count:", metadata.get("rowCount"))

Example 3: Chaining Discovery and Retrieval

# Workflow: enumerate all tables and inspect schema complexity

all_concepts = list_concepts()
table_ids = [c["id"] for c in all_concepts if c["type"] == "BigQuery Table"]

for cid in table_ids:
    try:
        md = read_concept_raw(cid)
        field_count = len(md.get("schema", []))
        print(f"{cid}: {field_count} fields")
    except ValueError:
        print(f"Concept {cid} not found")

Source Code Architecture

Understanding the functional split requires examining these key files in the repository:

• okf/src/reference_agent/tools/source_tools.py: Implements both list_concepts and read_concept_raw with their respective helper functions _ref_to_dict and parse_concept_id.

• okf/src/reference_agent/agent.py: Registers both tools with the LLM via FunctionTool wrappers, making them available for agent invocation.

• okf/src/reference_agent/sources/base.py: Defines the abstract interface requiring concrete sources to implement list_concepts() and read_concept(ref) methods.

• okf/src/reference_agent/sources/bigquery.py: Provides the concrete BigQuery implementation that returns table and view references during enumeration and extracts full table metadata during retrieval.

Summary

• list_concepts is an enumeration operation designed for catalog browsing and discovery, returning lightweight summaries of all available assets.

• read_concept_raw is a detail lookup operation that requires a specific concept_id and returns complete structured metadata for that single asset.

• Both tools access the active source configured in the agent context (get_context().source) and delegate to source-specific implementations.

• The tools are designed to be used sequentially: use list_concepts to discover asset identifiers, then read_concept_raw to inspect specific assets of interest.

• Error handling differs significantly: list_concepts returns empty lists when no assets exist, while read_concept_raw raises ValueError for invalid concept identifiers.

Frequently Asked Questions

Can I call read_concept_raw without using list_concepts first?

Yes, provided you know the valid concept_id string. The identifier follows a slash-joined format (e.g., "tables/dataset_name_table_name") that the source can resolve. However, since these identifiers are source-specific, using list_concepts first is the recommended approach to ensure valid identifiers.

What specific metadata fields does read_concept_raw return for BigQuery tables?

According to the BigQuery source implementation in okf/src/reference_agent/sources/bigquery.py, the returned dictionary includes schema (list of field definitions), partitioning (time or range partitioning config), clustering (clustering column names), rowCount, creationTime, lastModifiedTime, and other storage statistics. The exact fields depend on the source type.

Why does list_concepts return dictionaries instead of raw ConceptRef objects?

The _ref_to_dict helper in source_tools.py serializes ConceptRef objects into plain dictionaries to ensure compatibility with LLM function calling interfaces. This transformation strips internal Python objects while preserving the essential id, type, resource, and hint fields needed for asset identification.

Where is the error handling implemented when a concept is not found?

The read_concept_raw function in okf/src/reference_agent/tools/source_tools.py explicitly checks if src.find(cid) returns a valid reference. If the lookup returns None or fails to locate the concept, the function raises a ValueError with a descriptive message indicating that the concept_id could not be found in the active source.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}