How to Implement Fine-Grained Metric Logging per Training Step with CMF

The CMF library provides a two-phase API where log_metric() accumulates step-level data in memory during training, and commit_metrics() serializes the series to versioned Parquet files with full DVC and MLMD provenance.

The Hewlett Packard Enterprise Collaborative Machine-Learning Framework (CMF) enables detailed experiment tracking through its step-wise metric logging capabilities. Unlike frameworks that force immediate I/O on every step, CMF uses an in-memory buffering strategy that eliminates overhead in tight training loops while ensuring reproducible storage. This guide demonstrates how to use the log_metric() and commit_metrics() APIs from cmflib/cmf.py to capture training metrics at every epoch or iteration.

Understanding the CMF Metric Logging Architecture

CMF exposes two complementary methods in the Cmf class for fine-grained metric logging. The accumulation phase stores dictionaries in memory, while the commit phase handles serialization and provenance tracking.

In-Memory Accumulation with log_metric

The log_metric() method (implemented in cmflib/cmf.py at lines 1200–1226) accepts a name parameter and a dictionary of custom properties. Each call appends a new entry to self.metrics[name] using an incrementing sequence number as the key, preserving execution order without filesystem I/O.

Persistent Storage with commit_metrics

When training completes, commit_metrics() (lines 1248–1290 in cmflib/cmf.py) converts the accumulated dictionary into a pandas DataFrame via DataFrame.from_dict(..., orient="index"). It writes this to a Parquet file under the CMF artifacts directory hierarchy (self.ARTIFACTS_PATH/.../metrics), then registers the file with DVC using commit_output and captures the hash and URL. Finally, it creates a Step_Metrics artifact in the ML Metadata (MLMD) store, linking the metrics to the current execution and context.

Implementing Fine-Grained Metric Logging

Basic Per-Step Logging

Log metrics at each epoch without I/O overhead by accumulating data in memory:

from cmflib.cmf import Cmf

# Initialize CMF context

metawriter = Cmf()

# Training loop

for epoch in range(1, 11):
    # Compute metrics

    train_loss = 0.42 - (epoch * 0.01)
    train_acc = 0.85 + (epoch * 0.01)
    
    # Accumulate in memory (no I/O)

    metawriter.log_metric(
        "training_metrics",
        {"epoch": epoch, "loss": train_loss, "accuracy": train_acc}
    )

# Persist after loop completion

metawriter.commit_metrics("training_metrics")

The log_metric calls populate self.metrics["training_metrics"] with sequence-indexed entries. As shown in the official example at examples/example-get-started/src/train.py (lines 58–73), this pattern separates high-frequency metric capture from storage operations.

Distributed Training with Ray Tune

For hyperparameter tuning with Ray, use CmfRayLogger to maintain isolated contexts per trial:

from cmflib.cmf_ray_logger import CmfRayLogger

def train_with_ray(config):
    trial_id = config["trial_id"]
    cmf_logger = CmfRayLogger(trial_id)
    
    for step in range(config["max_steps"]):
        loss = compute_loss(step)
        # Log per-step metrics

        cmf_logger.cmf_obj[trial_id].log_metric(
            metrics_name="ray_training_metrics",
            custom_properties={"step": step, "loss": loss}
        )
    
    # Commit trial metrics

    cmf_logger.cmf_obj[trial_id].commit_metrics("ray_training_metrics")

This approach, detailed in cmflib/cmf_ray_logger.py (lines 66–78), maintains fine-grained metrics for parallel training runs while preserving DVC versioning and MLMD lineage per trial.

Retrieving Historical Metrics

Access stored step metrics by loading the Parquet file back into a DataFrame:

from cmflib.cmf import Cmf

cmf = Cmf()
metrics_df = cmf.read_metrics("training_metrics")
print(metrics_df.head())

The read_metrics method locates the Parquet file created by commit_metrics and returns the full step-wise history for analysis or visualization.

Summary

• Zero-overhead accumulation: log_metric() stores step data in a Python dictionary during training loops, avoiding per-step I/O bottlenecks.
• Atomic commits: commit_metrics() serializes accumulated metrics to Parquet, registers with DVC for Git-tracked versioning, and creates MLMD artifacts for lineage.
• Framework integration: Use CmfRayLogger for automatic per-trial context isolation in Ray Tune distributed workflows.
• Reproducible queries: Retrieve historical metrics via read_metrics() as pandas DataFrames for downstream analysis.

Frequently Asked Questions

What is the performance impact of logging metrics every training step?

CMF's design ensures zero filesystem I/O during log_metric() calls, as data accumulates only in self.metrics memory. The expensive serialization, DVC hashing, and MLMD registration occur only once when commit_metrics() is called, typically after epoch completion or at training end.

How does CMF handle metrics from distributed training runs?

The CmfRayLogger class in cmflib/cmf_ray_logger.py creates isolated CMF contexts per Ray trial, allowing each worker to maintain independent metric buffers. Each trial calls commit_metrics() independently, producing separate versioned Parquet files with unique DVC hashes and MLMD execution links.

Can I log non-scalar metrics like confusion matrices or embeddings?

The log_metric() API accepts any dictionary of custom properties, but commit_metrics() specifically converts the accumulated data to a pandas DataFrame for Parquet serialization. For complex artifacts like confusion matrices, use CMF's artifact logging APIs instead of the step-wise metric buffer.

Where are the committed metric files stored?

Parquet files are written to the CMF artifacts directory hierarchy under self.ARTIFACTS_PATH/.../metrics as implemented in the commit_metrics method. DVC tracks these files via commit_output, and MLMD stores the file hash and URL as a Step_Metrics artifact property, enabling retrieval through read_metrics().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}