How the Discovery Agent Performs Semantic Decomposition of Complex Search Queries

The Discovery Agent uses a Gemini-3-Flash LLM to break natural language queries into structured search variations, extracting predicates and generating up to three distinct query interpretations that target the Knowledge Catalog API.

The Google Cloud Platform Knowledge Catalog Discovery Agent translates ambiguous business questions into precise technical search queries through a process called semantic decomposition. Built on the Google ADK (Agent Development Kit) framework and residing in the GoogleCloudPlatform/knowledge-catalog repository, this agent enables users to search metadata using natural language instead of rigid query syntax. The transformation logic is defined in samples/discovery/SKILL.md and executed by the LLM agent defined in samples/discovery/agent.py.

Understanding the Discovery Agent Architecture

The Discovery Agent is not a simple keyword mapper. It is a reasoning agent that interprets intent, maps business terminology to technical metadata, and executes parallel search strategies to maximize recall.

According to the source code in samples/discovery/agent.py, the agent initializes as an llm_agent.Agent instance configured with the Gemini-3-Flash model. It loads its instruction set—which contains the semantic decomposition rules—from SKILL.md via the load_instruction() helper. The agent registers a single tool, knowledge_catalog_search, implemented in samples/discovery/tools.py, which forwards generated queries to the Knowledge Catalog backend.

The Semantic Decomposition Pipeline

When the Discovery Agent receives a user query, it executes a five-step decomposition process defined in the skill instructions.

Step 1: Query Classification and Preservation

First, the agent classifies the input as either free-text or predicate-bearing. If the user already includes explicit filters (e.g., type=table or system=bigquery), these predicates are preserved unchanged and embedded directly into the final query string. The agent isolates the semantic intent from any technical constraints already present.

Step 2: Entity and Constraint Extraction

The LLM parses the natural language request to identify core entities, metrics, and constraints. For a query like "show me revenue per month for the last quarter", the agent extracts entities such as "revenue" and "month," while identifying temporal constraints like "last quarter." This extraction follows the "Think Like a Data Engineer" principle, where business-level wording is mapped to underlying data model concepts (BigQuery tables, datasets, columns).

Step 3: Generating Multi-Strategy Variations

To ensure comprehensive coverage, the agent generates up to three distinct query variations that represent different search strategies:

• Variation 1 — Direct & Synonyms: Uses the user's exact terminology plus domain-specific synonyms (e.g., name:revenue or name:billing).
• Variation 2 — Data-Source Translation: Rewrites business concepts into technical terminology (e.g., mapping "customer acquisition" to system=bigquery AND type=table).
• Variation 3 — Broader System/Category: Casts a wider net by referencing related categories or higher-level business domains (e.g., system=bigquery AND (type=table OR type=dataset)).

All variations are constructed without double quotes, because the Knowledge Catalog Search API does not handle quoted phrases correctly.

Step 4: Predicate Mapping and Query Construction

Using the Predicate Reference Table defined in SKILL.md, the agent maps extracted keywords to formal search predicates (type, system, projectid, etc.). It constructs logical expressions using allowed operators (=, :, !=) and proper AND/OR grouping with parentheses.

Rule #1 in the skill file mandates that if a projectid is provided, it must be embedded directly into the query string to ensure proper scoping and access control.

Executing the Search Strategy

Once the decomposition phase generates the baseline query and its variations, the agent moves to execution and result processing.

Parallel Batch Search Execution

The agent issues a baseline search containing the full original user phrase alongside each generated variation. These calls execute in parallel via the knowledge_catalog_search tool. Every call includes the extracted predicates, ensuring that required project scoping (via projectid) is respected across all variations.

Result Deduplication and Ranking

Results from all parallel calls are merged, and duplicate entries (identified by matching display_name values) are collapsed into single entries. The agent then evaluates relevance by comparing entry metadata against the original semantic intent, sorts the list so the most relevant items appear first, and returns only the entry names without explanatory text.

Implementation Example

The following Python code demonstrates how to invoke the agent and the internal structure of the skill file that guides the LLM:

from samples.discovery.agent import discovery_agent

# Example user request requiring semantic decomposition

user_query = "show me the total revenue per month for the last quarter in project my-proj"

# The agent internally executes the SKILL.md decomposition logic

response = discovery_agent.run(user_query)

print(response)   # => List of Knowledge Catalog entry names

The skill file (samples/discovery/SKILL.md) instructs the LLM to generate variations using this pattern:

- **Variation 1 (Direct & Synonyms)**
  system=bigquery AND type=table AND name:revenue
- **Variation 2 (Data Source Translation)**
  system=bigquery AND type=table AND description:monthly
- **Variation 3 (Broader Category)**
  system=bigquery AND (type=table OR type=dataset) AND projectid=my-proj

The LLM applies these templates automatically based on the entities extracted during the decomposition phase.

Key Source Files

• samples/discovery/agent.py: Instantiates the Gemini-based LLM agent and loads the skill file via load_instruction().
• samples/discovery/SKILL.md: Contains the full instruction set, predicate reference tables, and semantic decomposition rules.
• samples/discovery/tools.py: Implements the knowledge_catalog_search tool that forwards generated queries to the Knowledge Catalog API.
• samples/discovery/utils.py: Provides helper functions to obtain the consumer project ID used for the Gemini model path.

Summary

• The Discovery Agent uses the Google ADK framework to perform semantic decomposition of complex natural language queries into executable Knowledge Catalog searches.
• Decomposition involves classifying query types, extracting entities and constraints, and applying "Think Like a Data Engineer" logic to map business terms to technical metadata.
• The agent generates up to three variations (Direct/Synonyms, Data-Source Translation, and Broader Category) to maximize search coverage without using double quotes.
• Predicate extraction relies on a reference table to build logical expressions with operators (=, :, !=) and mandatory projectid embedding for access control.
• Execution happens in parallel across baseline and variation queries, followed by deduplication and relevance-based ranking before returning entry names.

Frequently Asked Questions

How does the Discovery Agent handle queries that already contain technical predicates?

The agent first classifies the input to detect explicit predicates (e.g., type=table or system=bigquery). These existing predicates are preserved unchanged and embedded directly into the final query string, while the remaining free-text portion undergoes semantic decomposition to extract additional entities and constraints.

What are the three types of query variations generated during semantic decomposition?

The LLM generates Variation 1 (Direct & Synonyms) using exact terms and domain synonyms, Variation 2 (Data-Source Translation) rewriting business concepts into technical metadata terms, and Variation 3 (Broader System/Category) expanding the search to related categories or higher-level domains to ensure comprehensive recall.

Why does the Discovery Agent avoid double quotes in generated queries?

According to the implementation in samples/discovery/SKILL.md, Knowledge Catalog Search does not handle double quotes correctly in query strings. The agent constructs all search variations using unquoted text and explicit predicate operators to ensure compatibility with the backend API.

How is the project ID enforced during semantic decomposition?

Rule #1 in the skill file mandates that if a user provides a projectid, it must be embedded directly into every generated query string. This ensures that all parallel search calls respect project-level access controls and scoping requirements, preventing unauthorized cross-project data discovery.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}