# How Python Environment Tracking Ensures Reproducibility in the CMF Library

> Learn how Python environment tracking in the CMF library captures package specs, versions them with DVC, and links to Git commits to guarantee experiment reproducibility and enable exact reconstruction.

- Repository: [Hewlett Packard Enterprise/cmf](https://github.com/hewlettpackard/cmf)
- Tags: how-to-guide
- Published: 2026-03-03

---

**Python environment tracking in the CMF library captures exact Conda or pip package specifications, versions them with DVC, links them to Git commits, and stores them as Environment artifacts in the ML Metadata store to enable exact reconstruction of any historical experiment.**

The Common Modeling Framework (CMF) by Hewlett Packard provides systematic **Python environment tracking** to solve the "it works on my machine" problem in machine learning pipelines. By automatically recording every dependency version and channel configuration alongside code and data lineage, CMF creates an immutable provenance chain that makes any experiment reproducible across different machines and time periods.

## Core Mechanism of Python Environment Tracking

### Detecting the Active Runtime

In [`cmflib/utils/helper_functions.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/utils/helper_functions.py), the `get_python_env()` function inspects the current runtime to determine whether the active environment is managed by Conda or pip. If `CONDA_PREFIX` is detected, the function harvests the Conda package list, pip packages for hybrid environments, and channel URLs. For pure pip environments, it falls back to a flat list of installed packages. This detection logic resides at lines 54–96 and returns a structured dictionary containing the complete environment specification.

### Versioning Environment Files with DVC

Once the environment description is written to a file (typically [`environment.yaml`](https://github.com/hewlettpackard/cmf/blob/main/environment.yaml)), the `commit_output()` function in [`cmflib/dvc_wrapper.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/dvc_wrapper.py) (lines 45–86) executes `dvc add` to generate a content-addressable hash. This DVC hash acts as a cryptographic fingerprint of the exact dependency state. The function stages the resulting `.dvc` file in Git and returns the SHA-256 hash, ensuring that any subsequent modification to the environment file produces a new unique identifier.

### Capturing Git Provenance

To anchor the environment to specific source code, the `git_get_repo()` function in [`cmflib/dvc_wrapper.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/dvc_wrapper.py) (lines 90–107) retrieves the remote repository URL, while `commit_output()` simultaneously captures the current Git commit hash. These values establish an unambiguous link between the dependency snapshot and the exact code version used during execution.

### Creating the Environment Artifact

The `log_python_env()` method in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py) (implementation spanning lines 78–84 and the broader block around 580–650) constructs a first-class ML Metadata artifact of type "Environment". This artifact stores the DVC hash as the URI, the Git remote URL, the commit hash, and the logical DVC path. By treating the environment as a formal artifact rather than incidental metadata, CMF enables systematic querying and lineage tracking.

### Linking to the Execution Graph

Finally, `create_env_node()` in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py) (lines 58–71) registers the Environment artifact as a node in the graph backend (typically Neo4j). This creates an explicit edge connecting the Execution node to the Environment node, completing the provenance graph that relates code, data, and runtime configuration.

## The Reproducibility Chain

This multi-layered approach guarantees reproducibility through four immutable links:

- **Deterministic Package Lists**: Conda environment files include exact version pins and channel URLs, while pip requirements capture precise package states, enabling bitwise-identical environment reconstruction using `conda env create` or `pip install`.
- **Content-Addressable Storage**: DVC's SHA-256 hashes ensure that the retrieved environment file is byte-for-byte identical to the original, detecting any corruption or tampering.
- **Git Commit Anchoring**: The stored commit hash allows immediate checkout of the exact source code state that produced the artifact, eliminating code drift as a variable.
- **Graph-Based Lineage**: The Neo4j execution graph records which specific environment was used for which run, enabling queries like "find all models trained with TensorFlow 2.8.0."

## Practical Implementation Examples

### Automatic Environment Capture

The simplest way to leverage Python environment tracking is through the CMF client API:

```python
from cmflib.cmf import CMF

# Initialize the CMF client for your project

client = CMF(project_name="production_model")

# Create a new execution context

execution = client.create_execution(label="train_neural_network")

# Automatically detect, version, and log the current Python environment

client.log_python_env("environment.yaml")

```

This single call to `log_python_env()` triggers the complete workflow: environment detection via `get_python_env()`, DVC versioning via `commit_output()`, and artifact registration with graph linkage via `create_env_node()`.

### Manual Environment Serialization

For custom workflows, you can generate the environment file explicitly before logging:

```python
from cmflib.utils.helper_functions import get_python_env
import yaml

# Capture the current environment state

env_spec = get_python_env(env_name="training_env")

# Write to a file for inspection or manual editing

with open("environment.yaml", "w") as f:
    yaml.safe_dump(env_spec, f)

# Later, register this file with the metadata store

client.log_python_env("environment.yaml")

```

### Reconstructing a Historical Environment

To reproduce an experiment from an existing Environment artifact stored as `dvc://my_repo/env.yaml#abcd1234`:

```bash

# Retrieve the versioned environment file

dvc pull env.yaml.dvc

# Recreate the exact Conda environment

conda env create -f environment.yaml

# Or for pip-only environments, extract and install dependencies

pip install -r <(python -c "import yaml; d=yaml.safe_load(open('environment.yaml')); print('\n'.join(d.get('pip',[])))")

# Checkout the exact code version recorded in the artifact metadata

git checkout <Commit_hash>

```

## Summary

- **Python environment tracking** in CMF is implemented through the `get_python_env()` utility in [`cmflib/utils/helper_functions.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/utils/helper_functions.py), which detects Conda or pip environments and exports complete dependency specifications.
- The `commit_output()` function in [`cmflib/dvc_wrapper.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/dvc_wrapper.py) versions these specifications using DVC content-addressable hashing, while `git_get_repo()` captures source code provenance.
- `log_python_env()` in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py) creates formal Environment artifacts in the ML Metadata store and links them to execution nodes via `create_env_node()`.
- This architecture enables exact reproduction of any historical experiment by combining package snapshots, cryptographic versioning, Git anchoring, and graph-based lineage tracking.

## Frequently Asked Questions

### What file formats does CMF use to store Python environment tracking data?

CMF generates standard Conda [`environment.yaml`](https://github.com/hewlettpackard/cmf/blob/main/environment.yaml) files for Conda environments and pip-compatible requirements lists for pure pip environments. These are plain text files that can be read by humans and consumed by standard package managers, stored as artifacts in the metadata store with DVC hashes for integrity verification.

### How does CMF handle both Conda and pip environments?

The `get_python_env()` function in [`cmflib/utils/helper_functions.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/utils/helper_functions.py) automatically detects which package manager is active by checking for the `CONDA_PREFIX` environment variable. If present, it captures both Conda packages and any pip-installed packages within that Conda environment; otherwise, it captures the global pip package list, ensuring comprehensive coverage regardless of the user's setup.

### Can I reproduce an environment without access to the original Git repository?

While the DVC hash ensures you have the correct environment file, reproducing the complete experiment requires access to the Git remote stored in the artifact metadata. Without the original repository, you cannot checkout the specific commit hash recorded in the Environment artifact, though you can still reconstruct the Python environment itself using the dependency file.

### Where is the environment metadata stored in the CMF architecture?

Environment metadata is stored as a first-class artifact in the ML Metadata (MLMD) store, managed through the `log_python_env()` method in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py). Additionally, when graph tracking is enabled, relationships between executions and environments are stored in the configured graph database (such as Neo4j) via the `create_env_node()` function, enabling complex lineage queries.