How to Implement Experiment Tracking Using Hypster Cards

Hypster automatically tracks which configuration parameters are accessed during execution through the HP.called_params set, enabling effortless experiment reproducibility without modifying your existing config code.

Hypster, an open-source configuration framework maintained at gilad-rubin/hypster, provides built-in experiment tracking that captures exactly which parameters influence each run. Unlike external logging libraries that require manual instrumentation, Hypster records parameter access automatically through the HP class's internal state. This guide demonstrates how to leverage called_params and document your tracking implementation using Hypster's documentation cards.

How Hypster Tracks Parameter Access

The called_params Set Architecture

In src/hypster/hp.py, the HP class initializes called_params as an empty Python set in its constructor (line 55). This set stores the fully-qualified names of every parameter accessed during configuration execution.

Each parameter method—whether int(), float(), select(), or bool()—populates this set through low-level handlers like _execute_single and _handle_select_single (approximately lines 120 and 304). Before returning a value, these handlers calculate the complete parameter path using _get_full_param_path (lines 58-62) and add it to self.called_params when the track_called flag is enabled.

Because the tracking mechanism lives entirely in Python objects, you can export the data to JSON, CSV, or external experiment-tracking platforms like Weights & Biases or MLflow without altering your original configuration logic.

Propagation Through Nested Configurations

When using nested configurations, the child HP instance receives a direct reference to its parent's called_params set. At line 385 in src/hypster/hp.py, the nesting logic executes nested_hp.called_params = self.called_params, ensuring parameters accessed at any depth merge into the top-level set automatically.

This design produces dot-separated namespaces for nested parameters (e.g., sub.alpha), enabling hierarchical experiment tracking across complex configuration trees.

Accessing Experiment Tracking Data

Direct HP Instance Inspection

Access the tracking data by inspecting the called_params attribute after executing your configuration function:

from hypster import HP

# Initialize HP with intended values

hp = HP({"lr": 0.01, "batch": 64, "use_aug": False})

def model_cfg(hp: HP):
    lr = hp.float(0.001, name="lr", min=0, max=1)
    batch = hp.int(32, name="batch")
    aug = hp.bool(True, name="use_aug")
    return {"lr": lr, "batch": batch, "use_aug": aug}

# Execute the config

model_cfg(hp)

# Retrieve accessed parameters

print(hp.called_params)  # {'lr', 'batch', 'use_aug'}

The resulting set contains exactly the parameter names that influenced this specific run, which can be written to disk or sent to a monitoring service.

Integration with instantiate()

When using the higher-level API, wrap the call to capture tracking information:

from hypster import HP

def instantiate_with_tracking(func, *, values=None, **kwargs):
    hp = HP(values or {})
    result = func(hp, **kwargs)
    return result, hp.called_params

# Usage example

def train_cfg(hp: HP):
    lr = hp.float(0.001, name="lr")
    epochs = hp.int(10, name="epochs")
    return {"lr": lr, "epochs": epochs}

cfg, used = instantiate_with_tracking(
    train_cfg, 
    values={"lr": 0.01, "epochs": 100}
)
print("Tracked:", used)  # {'lr', 'epochs'}

The used set can be serialized with json.dumps(list(used)) for persistent storage alongside experiment artifacts.

Hierarchical Tracking in Nested Configs

Nested configurations automatically merge tracking data into the parent namespace. Consider this hierarchical setup:

def sub_cfg(hp: HP):
    alpha = hp.float(0.5, name="alpha")
    return {"alpha": alpha}

def main_cfg(hp: HP):
    beta = hp.int(10, name="beta")
    hp.nest(sub_cfg, name="sub")
    return {"beta": beta}

hp = HP({"beta": 20, "sub.alpha": 0.75})
main_cfg(hp)

print(hp.called_params)  # {'beta', 'sub.alpha'}

The namespace handling in _get_full_param_path (lines 58-62) ensures nested parameters are recorded with their full dot-separated paths, maintaining clear separation between configuration levels.

Surfacing Tracking Information with Hypster Cards

While the tracking implementation lives in Python code, you can document this feature for users through Hypster's documentation cards. The repository uses GitBook to render tables in docs/reproducibility/cards.md as visual cards.

To surface experiment tracking in the documentation:

1. Create a documentation file at docs/reproducibility/experiment-tracking.md containing your tracking examples and explanation.

2. Reference it in the cards table by adding a row to docs/reproducibility/cards.md:

| **Experiment Tracking** | Automatically capture which parameters were used in each run | | [experiment-tracking.md](experiment-tracking.md) |

GitBook will automatically render this as a clickable card, directing developers to your experiment tracking guide.

Complete End-to-End Implementation

Here is a full example combining nested configurations, tracking, and JSON export:

# file: experiment.py

from hypster import HP
import json
import pathlib

def data_cfg(hp: HP):
    path = hp.text("/data/train.csv", name="data_path")
    shuffle = hp.bool(True, name="shuffle")
    return {"path": path, "shuffle": shuffle}

def model_cfg(hp: HP):
    lr = hp.float(0.001, name="lr", min=0, max=1)
    hidden = hp.int(256, name="hidden")
    return {"lr": lr, "hidden": hidden}

def full_cfg(hp: HP):
    hp.nest(data_cfg, name="data")
    hp.nest(model_cfg, name="model")
    return {}

def run_experiment(values: dict):
    hp = HP(values)
    config = full_cfg(hp)
    return config, hp.called_params

# Execute with specific values

values = {
    "data.data_path": "/mnt/dataset.csv",
    "data.shuffle": False,
    "model.lr": 0.01,
    "model.hidden": 512,
}

config, tracked_params = run_experiment(values)

# Persist tracking data

pathlib.Path("run_01_metadata.json").write_text(
    json.dumps({
        "config": config,
        "parameters_used": list(tracked_params)
    }, indent=2)
)

print(f"Experiment used {len(tracked_params)} parameters: {tracked_params}")

This pattern ensures every experiment run records exactly the parameters that influenced execution, enabling full reproducibility and audit trails.

Summary

• Automatic discovery: The HP class in src/hypster/hp.py creates and populates called_params (line 55) without requiring manual instrumentation of your configuration code.
• Hierarchical coverage: Nested configurations share the parent's tracking set via reference assignment (line 385), capturing dot-separated paths like model.lr.
• Flexible export: The tracking set can be serialized to JSON or integrated with external platforms immediately after execution.
• Documentation integration: Use docs/reproducibility/cards.md to create visual documentation cards linking to your experiment tracking guide.

Frequently Asked Questions

What is called_params in Hypster?

called_params is a Python set attribute of the HP class that stores the fully-qualified names of every parameter accessed during configuration execution. It is initialized in the HP constructor at line 55 of src/hypster/hp.py and populated by internal handlers before parameter values are returned to your configuration function.

Does experiment tracking work with nested Hypster configurations?

Yes. When you call hp.nest(child_func, name="sub"), Hypster passes a reference to the parent's called_params set to the child HP instance (line 385 in src/hypster/hp.py). This ensures parameters accessed at any nesting level are automatically recorded with dot-separated namespaces like sub.param_name.

How do I integrate Hypster tracking with Weights & Biases or MLflow?

After executing your configuration function, simply convert hp.called_params to a list and log it alongside your metrics. For example: wandb.log({"config": config, "parameters_used": list(hp.called_params)}). The set contains exactly the parameter names that influenced the run, making it easy to compare which configurations were active across different experiments.

Where is the experiment tracking data stored?

The tracking data resides temporarily in the HP instance's memory as a Python set. It is not persisted automatically; you must export it after execution. Common patterns include writing to JSON files, logging to stdout, or transmitting to external experiment-tracking services before the hp variable goes out of scope.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}