# Architecture of the Metrics Catalog and Caching System in MCP Ambari API

> Explore the three-layer TTL-driven caching system architecture in the MCP Ambari API. Learn how it aggregates Ambari Metrics Service metadata into a dynamic, read-only catalog, preventing thundering herds with async locks.

- Repository: [JungJungIn/mcp-ambari-api](https://github.com/call518/mcp-ambari-api)
- Tags: architecture
- Published: 2026-02-26

---

**The architecture implements a three-layer, TTL-driven caching system that dynamically aggregates Ambari Metrics Service metadata into a read-only catalog protected by async locks to prevent thundering herds.**

The `call518/mcp-ambari-api` server maintains a dynamic metrics catalog that serves as the single source of truth for all metric discovery operations. Its architecture combines raw metadata caching, dynamic catalog generation, and synonym normalization to deliver low-latency responses without overwhelming the backing Ambari Metrics Service. This article examines the source code implementation to explain how the caching layers interact, how concurrency is managed, and how developers can interact with the catalog programmatically.

## Three-Layer Caching Architecture

The metrics catalog is built from three cooperating layers defined in [`src/mcp_ambari_api/functions.py`](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py). Each layer serves a distinct purpose in minimizing network overhead and normalizing user input.

### Layer 1: Raw AMS Metadata Cache

The **raw metadata cache** stores the unprocessed response from the Ambari Metrics Service `/metrics/metadata` endpoint. This avoids repeated network calls when multiple coroutines request metadata.

- **Implementation**: The global dictionary `_METRICS_METADATA_CACHE` maps a cache key—either a specific `appId` or the special key `"__all__"`—to a structure containing `{timestamp, entries}`.
- **TTL**: Cache entries expire based on the `AMBARI_METRICS_METADATA_TTL` environment variable, defaulting to **300 seconds** (5 minutes).
- **Location**: Lines [85‑91](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L85-L91).

### Layer 2: Dynamic Catalog Cache

The **dynamic catalog cache** collates raw metadata into two derived structures: a catalog mapping `appId` to sorted metric name lists, and a lookup table for case-insensitive app resolution.

- **Implementation**: `_DYNAMIC_CATALOG_CACHE` holds `{timestamp, catalog, lookup}`.
- **Refresh**: The `ensure_metric_catalog()` function rebuilds this structure when the TTL expires.
- **Concurrency**: Access is protected by the async lock `_catalog_lock` (defined at lines [118‑119](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L118-L119)) to ensure only one coroutine rebuilds the catalog at a time.
- **Location**: Lines [108‑112](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L108-L112).

### Layer 3: Synonym and Exclusion Logic

The top layer normalizes user-provided app names through synonyms and filters out internal collectors that should not be exposed.

- **Synonyms**: The `APP_SYNONYMS` dictionary (lines [95‑103](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L95-L103)) maps common aliases like `"nn"` to the canonical `"namenode"`.
- **Exclusions**: The `EXCLUDED_APP_IDS` set (lines [105‑107](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L105-L107)) prunes internal apps such as `amssmoketestfake` from the final catalog.
- **Helpers**: `_normalize_app_key()`, `_is_excluded_app()`, and `canonicalize_app_id()` (lines [124‑152](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L124-L152)) apply these rules during lookup.

## Catalog Construction Flow

The `ensure_metric_catalog()` function orchestrates the build process through a deterministic seven-step flow:

1. **Cache-hit fast path** – If `use_cache` is true and the catalog timestamp is younger than the TTL, the function returns the cached structures immediately (lines [46‑50](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L46-L50)).

2. **Lock acquisition** – On a cache miss, the coroutine acquires `_catalog_lock` to guarantee a single build and prevent thundering-herd scenarios (lines [52‑55](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L52-L55)).

3. **Metadata fetch** – The function calls `get_metrics_metadata(None)` to request metadata for all apps. If the response is empty, a fallback loop queries each `appId` in the curated list `CURATED_METRIC_APP_IDS` individually (lines [62‑73](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L62-L73)).

4. **Entry iteration** – For each metadata record, the code extracts `appId` and `metricName`, discarding excluded apps. It populates two structures:
   - `metrics_by_app[app] = set(metric names)` (line [91](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L91))
   - `lookup[lower(app)] = app` (line [92](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L92))

5. **Canonical guarantee** – The builder ensures every entry defined in `APP_SYNONYMS` has at least an empty metric list and a corresponding lookup entry (lines [97‑103](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L97-L103)).

6. **Exclusion pruning** – Any app appearing in `EXCLUDED_APP_IDS` is removed from the working set (lines [104‑108](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L104-L108)).

7. **Cache write** – The fresh catalog and lookup are stored in `_DYNAMIC_CATALOG_CACHE` with the current timestamp (lines [109‑113](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L109-L113)).

The final catalog is a **read-only dictionary** (`{appId: [sorted metric names]}`) that can be safely shared across many coroutines without additional synchronization.

## Concurrency Control and Thread Safety

The architecture uses Python's `asyncio` primitives to ensure safety in an async context. The `_catalog_lock` (an `asyncio.Lock`) serializes write access to `_DYNAMIC_CATALOG_CACHE`. 

When a cache miss occurs, subsequent requests for the catalog block on the lock until the building coroutine completes. Once the lock releases, all waiters receive the freshly built catalog instance. This pattern eliminates redundant network requests and CPU-intensive aggregation work during high-concurrency scenarios.

## Public API and Usage Examples

The module exposes several high-level helpers that honor the `use_cache` flag, allowing callers to force refresh after cluster configuration changes.

```python

# Retrieve the full catalog (cached)

catalog, lookup = await ensure_metric_catalog()
print("First 5 apps:", list(catalog.keys())[:5])

# List all canonical app IDs

app_ids = await get_available_app_ids()
print("Total apps:", len(app_ids))

# Resolve metrics for a user-provided identifier (synonyms supported)

metrics = await get_metrics_for_app("nn")  # Resolves to "namenode"

print("Sample metrics:", metrics[:5])

# Force a catalog refresh (bypass cache)

fresh_catalog, _ = await ensure_metric_catalog(use_cache=False)

```

- `get_metric_catalog()` (lines [154‑155](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L154-L155)) returns the full catalog dictionary.
- `get_available_app_ids()` (lines [159‑161](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L159-L161)) lists all canonical app IDs.
- `get_metrics_for_app(app_id)` (lines [164‑176](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L164-L176)) resolves synonyms via `canonicalize_app_id()` before lookup.
- `metric_supported_for_app(app_id, metric_name)` (lines [181‑185](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L181-L185)) checks membership efficiently.

## Configuration and Key Files

| File | Role | Key Components |
|------|------|----------------|
| [`src/mcp_ambari_api/functions.py`](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py) | Core caching and catalog logic | `_METRICS_METADATA_CACHE`, `_DYNAMIC_CATALOG_CACHE`, `ensure_metric_catalog`, synonym logic |
| [`src/mcp_ambari_api/mcp_main.py`](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/mcp_main.py) | MCP tool registration | Exposes `get_metric_catalog`, `get_metrics_for_app` to LLM agents |
| `.env.example` | Runtime configuration | Defines `AMBARI_METRICS_METADATA_TTL` and exclusion lists |

All code references point to the implementation in `call518/mcp-ambari-api` as of the main branch.

## Summary

- The architecture uses **three distinct layers**: raw AMS metadata caching, dynamic catalog aggregation, and synonym/exclusion normalization.
- **TTL-driven expiration** (default 300s) balances freshness with network efficiency.
- **Async locks** prevent thundering-herd rebuilds during cache misses.
- The catalog is **read-only after construction**, allowing lock-free concurrent reads.
- **Synonym resolution** enables user-friendly identifiers like `"nn"` while maintaining canonical `"namenode"` keys internally.

## Frequently Asked Questions

### How does the caching system prevent stale data?

The system timestamps every cache entry and validates age against `AMBARI_METRICS_METADATA_TTL` (default 300 seconds) on each access. Callers can force immediate refresh by passing `use_cache=False` to `ensure_metric_catalog()` or related helpers, which bypasses the timestamp check and triggers a fresh fetch from the Ambari Metrics Service.

### What happens when two requests arrive simultaneously while the cache is empty?

The first coroutine to arrive acquires the `_catalog_lock` (lines [118‑119](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L118-L119)) and proceeds to build the catalog. Subsequent coroutines block on the same lock until construction completes, at which point they all receive the newly cached read-only dictionary. This guarantees exactly one network fetch and one aggregation pass per TTL window.

### Can I customize which applications appear in the catalog?

Yes. Populate the `EXCLUDED_APP_IDS` environment variable or modify the `EXCLUDED_APP_IDS` set in [`src/mcp_ambari_api/functions.py`](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py) (lines [105‑107](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L105-L107)) to filter specific `appId` values. Additionally, you can extend `APP_SYNONYMS` (lines [95‑103](https://github.com/call518/mcp-ambari-api/blob/main/src/mcp_ambari_api/functions.py#L95-L103)) to add custom aliases for your organization's naming conventions.

### Is the metrics catalog shared across all MCP tools?

Yes. The global dictionaries `_METRICS_METADATA_CACHE` and `_DYNAMIC_CATALOG_CACHE` are module-level singletons. All coroutines within the same process share these instances, ensuring consistent state across tools like `get_metric_catalog` and `get_metrics_for_app` defined in [`mcp_main.py`](https://github.com/call518/mcp-ambari-api/blob/main/mcp_main.py).