Architecture of the Metrics Catalog and Caching System in MCP Ambari API

The architecture implements a three-layer, TTL-driven caching system that dynamically aggregates Ambari Metrics Service metadata into a read-only catalog protected by async locks to prevent thundering herds.

The call518/mcp-ambari-api server maintains a dynamic metrics catalog that serves as the single source of truth for all metric discovery operations. Its architecture combines raw metadata caching, dynamic catalog generation, and synonym normalization to deliver low-latency responses without overwhelming the backing Ambari Metrics Service. This article examines the source code implementation to explain how the caching layers interact, how concurrency is managed, and how developers can interact with the catalog programmatically.

Three-Layer Caching Architecture

The metrics catalog is built from three cooperating layers defined in src/mcp_ambari_api/functions.py. Each layer serves a distinct purpose in minimizing network overhead and normalizing user input.

Layer 1: Raw AMS Metadata Cache

The raw metadata cache stores the unprocessed response from the Ambari Metrics Service /metrics/metadata endpoint. This avoids repeated network calls when multiple coroutines request metadata.

• Implementation: The global dictionary _METRICS_METADATA_CACHE maps a cache key—either a specific appId or the special key "__all__"—to a structure containing {timestamp, entries}.
• TTL: Cache entries expire based on the AMBARI_METRICS_METADATA_TTL environment variable, defaulting to 300 seconds (5 minutes).
• Location: Lines 85‑91.

Layer 2: Dynamic Catalog Cache

The dynamic catalog cache collates raw metadata into two derived structures: a catalog mapping appId to sorted metric name lists, and a lookup table for case-insensitive app resolution.

• Implementation: _DYNAMIC_CATALOG_CACHE holds {timestamp, catalog, lookup}.
• Refresh: The ensure_metric_catalog() function rebuilds this structure when the TTL expires.
• Concurrency: Access is protected by the async lock _catalog_lock (defined at lines 118‑119) to ensure only one coroutine rebuilds the catalog at a time.
• Location: Lines 108‑112.

Layer 3: Synonym and Exclusion Logic

The top layer normalizes user-provided app names through synonyms and filters out internal collectors that should not be exposed.

• Synonyms: The APP_SYNONYMS dictionary (lines 95‑103) maps common aliases like "nn" to the canonical "namenode".
• Exclusions: The EXCLUDED_APP_IDS set (lines 105‑107) prunes internal apps such as amssmoketestfake from the final catalog.
• Helpers: _normalize_app_key(), _is_excluded_app(), and canonicalize_app_id() (lines 124‑152) apply these rules during lookup.

Catalog Construction Flow

The ensure_metric_catalog() function orchestrates the build process through a deterministic seven-step flow:

1. Cache-hit fast path – If use_cache is true and the catalog timestamp is younger than the TTL, the function returns the cached structures immediately (lines 46‑50).

2. Lock acquisition – On a cache miss, the coroutine acquires _catalog_lock to guarantee a single build and prevent thundering-herd scenarios (lines 52‑55).

3. Metadata fetch – The function calls get_metrics_metadata(None) to request metadata for all apps. If the response is empty, a fallback loop queries each appId in the curated list CURATED_METRIC_APP_IDS individually (lines 62‑73).

4. Entry iteration – For each metadata record, the code extracts appId and metricName, discarding excluded apps. It populates two structures:

   • metrics_by_app[app] = set(metric names) (line 91)
   • lookup[lower(app)] = app (line 92)

5. Canonical guarantee – The builder ensures every entry defined in APP_SYNONYMS has at least an empty metric list and a corresponding lookup entry (lines 97‑103).

6. Exclusion pruning – Any app appearing in EXCLUDED_APP_IDS is removed from the working set (lines 104‑108).

7. Cache write – The fresh catalog and lookup are stored in _DYNAMIC_CATALOG_CACHE with the current timestamp (lines 109‑113).

The final catalog is a read-only dictionary ({appId: [sorted metric names]}) that can be safely shared across many coroutines without additional synchronization.

Concurrency Control and Thread Safety

The architecture uses Python's asyncio primitives to ensure safety in an async context. The _catalog_lock (an asyncio.Lock) serializes write access to _DYNAMIC_CATALOG_CACHE.

When a cache miss occurs, subsequent requests for the catalog block on the lock until the building coroutine completes. Once the lock releases, all waiters receive the freshly built catalog instance. This pattern eliminates redundant network requests and CPU-intensive aggregation work during high-concurrency scenarios.

Public API and Usage Examples

The module exposes several high-level helpers that honor the use_cache flag, allowing callers to force refresh after cluster configuration changes.

# Retrieve the full catalog (cached)

catalog, lookup = await ensure_metric_catalog()
print("First 5 apps:", list(catalog.keys())[:5])

# List all canonical app IDs

app_ids = await get_available_app_ids()
print("Total apps:", len(app_ids))

# Resolve metrics for a user-provided identifier (synonyms supported)

metrics = await get_metrics_for_app("nn")  # Resolves to "namenode"

print("Sample metrics:", metrics[:5])

# Force a catalog refresh (bypass cache)

fresh_catalog, _ = await ensure_metric_catalog(use_cache=False)

• get_metric_catalog() (lines 154‑155) returns the full catalog dictionary.
• get_available_app_ids() (lines 159‑161) lists all canonical app IDs.
• get_metrics_for_app(app_id) (lines 164‑176) resolves synonyms via canonicalize_app_id() before lookup.
• metric_supported_for_app(app_id, metric_name) (lines 181‑185) checks membership efficiently.

Configuration and Key Files

File	Role	Key Components
src/mcp_ambari_api/functions.py	Core caching and catalog logic	_METRICS_METADATA_CACHE, _DYNAMIC_CATALOG_CACHE, ensure_metric_catalog, synonym logic
src/mcp_ambari_api/mcp_main.py	MCP tool registration	Exposes get_metric_catalog, get_metrics_for_app to LLM agents
.env.example	Runtime configuration	Defines AMBARI_METRICS_METADATA_TTL and exclusion lists

All code references point to the implementation in call518/mcp-ambari-api as of the main branch.

Summary

• The architecture uses three distinct layers: raw AMS metadata caching, dynamic catalog aggregation, and synonym/exclusion normalization.
• TTL-driven expiration (default 300s) balances freshness with network efficiency.
• Async locks prevent thundering-herd rebuilds during cache misses.
• The catalog is read-only after construction, allowing lock-free concurrent reads.
• Synonym resolution enables user-friendly identifiers like "nn" while maintaining canonical "namenode" keys internally.

Frequently Asked Questions

How does the caching system prevent stale data?

The system timestamps every cache entry and validates age against AMBARI_METRICS_METADATA_TTL (default 300 seconds) on each access. Callers can force immediate refresh by passing use_cache=False to ensure_metric_catalog() or related helpers, which bypasses the timestamp check and triggers a fresh fetch from the Ambari Metrics Service.

What happens when two requests arrive simultaneously while the cache is empty?

The first coroutine to arrive acquires the _catalog_lock (lines 118‑119) and proceeds to build the catalog. Subsequent coroutines block on the same lock until construction completes, at which point they all receive the newly cached read-only dictionary. This guarantees exactly one network fetch and one aggregation pass per TTL window.

Can I customize which applications appear in the catalog?

Yes. Populate the EXCLUDED_APP_IDS environment variable or modify the EXCLUDED_APP_IDS set in src/mcp_ambari_api/functions.py (lines 105‑107) to filter specific appId values. Additionally, you can extend APP_SYNONYMS (lines 95‑103) to add custom aliases for your organization's naming conventions.

Is the metrics catalog shared across all MCP tools?

Yes. The global dictionaries _METRICS_METADATA_CACHE and _DYNAMIC_CATALOG_CACHE are module-level singletons. All coroutines within the same process share these instances, ensuring consistent state across tools like get_metric_catalog and get_metrics_for_app defined in mcp_main.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}