# How to Handle Column-Level Lineage for Metrics in Ossie

> Learn how Ossie handles column-level lineage for metrics by parsing SQL expressions and mapping columns to their originating field definitions in the semantic model.

- Repository: [The Apache Software Foundation/ossie](https://github.com/apache/ossie)
- Tags: how-to-guide
- Published: 2026-07-17

---

**OSSIE derives column-level lineage for metrics by parsing the SQL expressions stored in `OSIDialectExpression` objects and mapping referenced columns back to their originating `OSIField` definitions within the semantic model.**

Apache OSSIE (Open Source Semantic Information Exchange) models metrics as first-class objects that encapsulate dialect-specific SQL expressions, enabling precise tracking of data lineage from calculated metrics back to source dataset columns. By examining these expressions and resolving column references against the semantic model's field definitions, you can build complete column-level lineage graphs for downstream data governance platforms like Snowflake or OrionBelt.

## Understanding OSSIE's Metric Architecture

OSSIE represents metrics through a hierarchical model defined in [`python/src/ossie/models.py`](https://github.com/apache/ossie/blob/main/python/src/ossie/models.py). This architecture separates the metric's identity from its implementation across different SQL dialects.

### OSIMetric and OSIExpression Structure

The **`OSIMetric`** class ([[`models.py`](https://github.com/apache/ossie/blob/main/models.py)](https://github.com/apache/ossie/blob/main/python/src/ossie/models.py#L39-L48)) serves as the container for metric definitions, holding the metric name and an associated **`OSIExpression`**. The `OSIExpression` class ([[`models.py`](https://github.com/apache/ossie/blob/main/models.py)](https://github.com/apache/ossie/blob/main/python/src/ossie/models.py#L80-L86)) encapsulates a collection of dialect-specific implementations, allowing the same logical metric to be rendered as Snowflake SQL, ANSI SQL, or MAQL depending on the target platform.

### Dialect-Specific Expression Storage

Each dialect implementation is stored as an **`OSIDialectExpression`** ([[`models.py`](https://github.com/apache/ossie/blob/main/models.py)](https://github.com/apache/ossie/blob/main/python/src/ossie/models.py#L71-L78)). This object pairs a dialect identifier (`OSIDialect`) with the raw expression string. Because OSSIE stores these expressions as strings rather than parsed ASTs, column-level lineage extraction requires external parsing to identify column references within the SQL text.

## Implementing Column-Level Lineage Extraction

The typical workflow to obtain column-level lineage involves defining metrics with explicit field references, parsing the SQL to extract identifiers, and mapping those identifiers back to the dataset fields.

### Step 1: Define Metrics with Fully-Qualified References

When creating an `OSIMetric`, use fully-qualified field names (e.g., `Orders.amount`) in your dialect expressions. This practice ensures that lineage extraction can unambiguously map columns to their source datasets.

```python
from ossie.models import OSIMetric, OSIExpression, OSIDialectExpression, OSIDialect

metric = OSIMetric(
    name="TotalSales",
    expression=OSIExpression(
        dialects=[
            OSIDialectExpression(
                dialect=OSIDialect.SNOWFLAKE,
                expression="SUM(Orders.amount * Orders.price)"
            ),
            OSIDialectExpression(
                dialect=OSIDialect.ANSI_SQL,
                expression="SUM(Orders.amount * Orders.price)"
            ),
        ]
    ),
    description="Total sales amount in monetary units",
)

```

### Step 2: Parse Dialect Expressions to Identify Columns

Because OSSIE does not enforce parsing internally, you must use a SQL parser—such as `sqlglot` via the Snowflake converter or another dialect-specific parser—to walk the expression AST and locate column references. The converters in [`converters/snowflake/src/osi_to_snowflake_yaml_converter.py`](https://github.com/apache/ossie/blob/main/converters/snowflake/src/osi_to_snowflake_yaml_converter.py) demonstrate how to transform dialect expressions while preserving lineage information.

### Step 3: Map Column Identifiers to OSIField Objects

Once you extract column identifiers from the parsed expression, resolve each reference to the corresponding **`OSIField`** object in the dataset. This resolution creates the lineage link from the metric back to the source columns.

```python
import sqlglot
from ossie.models import OSIDataset, OSIField, OSIMetric, OSIDialect

def extract_lineage(metric: OSIMetric, datasets: list[OSIDataset]) -> dict[str, list[OSIField]]:
    """Return a mapping of column names ↦ OSIField objects used by the metric."""
    lineage = {}
    # Choose a dialect that we can parse (e.g., SNOWFLAKE / ANSI_SQL)

    expr = next(
        d.expression
        for d in metric.expression.dialects
        if d.dialect in {OSIDialect.SNOWFLAKE, OSIDialect.ANSI_SQL}
    )
    # Parse the expression with sqlglot

    ast = sqlglot.parse_one(expr, read=metric.expression.dialects[0].dialect.lower())
    for col in ast.find_all(sqlglot.exp.Column):
        col_name = f"{col.table}.{col.name}"
        # Locate the field in the supplied datasets

        for ds in datasets:
            if ds.name.lower() == col.table.lower():
                field = next((f for f in ds.fields or [] if f.name == col.name), None)
                if field:
                    lineage.setdefault(col_name, []).append(field)
    return lineage

```

### Step 4: Serialize Lineage for Downstream Tools

After resolving the lineage mappings, serialize the complete semantic model—including datasets, fields, and metrics—to OSSIE-compatible YAML or JSON. This format can be consumed by downstream lineage tools or converted to platform-specific metadata.

```python
from ossie.models import OSIDocument

doc = OSIDocument(
    version="0.2.0.dev0",
    semantic_model=[
        # ... include datasets, relationships, and the metric defined above ...

    ],
)

print(doc.to_osi_yaml())

```

## Validation and Converter Integration

The **[`validation/validate.py`](https://github.com/apache/ossie/blob/main/validation/validate.py)** module ensures that metric names are unique and that expressions are syntactically valid before lineage extraction begins. This validation step prevents errors during the parsing phase.

Converters such as the **Snowflake** converter and **OrionBelt** ([[`converters/orionbelt/README.md`](https://github.com/apache/ossie/blob/main/converters/orionbelt/README.md)](https://github.com/apache/ossie/blob/main/converters/orionbelt/README.md)) provide concrete implementations for parsing dialect-specific expressions. These converters walk the expression AST to locate column references and emit OSSIE-compatible lineage objects, ensuring that lineage information is retained when translating metrics across vendor boundaries.

## Summary

- **OSIMetric** objects contain **OSIExpression** instances that store raw SQL strings in **OSIDialectExpression** containers, enabling multi-dialect support.
- Column-level lineage requires parsing these expression strings using tools like `sqlglot` or converter-specific parsers to extract column references.
- Extracted column identifiers must be mapped to **OSIField** objects within the corresponding datasets to establish complete lineage paths.
- The [`validation/validate.py`](https://github.com/apache/ossie/blob/main/validation/validate.py) module ensures metric integrity before lineage extraction, while converters handle dialect-specific parsing and platform translation.
- Lineage metadata can be serialized to YAML/JSON via **OSIDocument** for integration with external data governance tools.

## Frequently Asked Questions

### What is the relationship between OSIMetric and OSIExpression?

An `OSIMetric` represents a single logical metric (such as "Total Revenue") and contains exactly one `OSIExpression` object. The `OSIExpression` acts as a container for multiple `OSIDialectExpression` instances, each representing the metric's implementation in a specific SQL dialect (Snowflake, ANSI SQL, MAQL, etc.). This separation allows the same metric definition to be deployed across different database platforms while maintaining a single source of truth for lineage purposes.

### How does OSSIE handle different SQL dialects when extracting lineage?

OSSIE stores dialect-specific SQL as raw strings within `OSIDialectExpression` objects tagged with an `OSIDialect` enum value. When extracting lineage, you select the appropriate dialect expression and use a parser compatible with that dialect—such as the Snowflake converter for Snowflake SQL or a generic SQL parser for ANSI SQL. The [`converters/snowflake/src/osi_to_snowflake_yaml_converter.py`](https://github.com/apache/ossie/blob/main/converters/snowflake/src/osi_to_snowflake_yaml_converter.py) implementation demonstrates how to parse these expressions while preserving column references for lineage tracking.

### Can lineage be extracted without using external parsers like sqlglot?

While OSSIE does not provide built-in SQL parsing, the raw expression strings in `OSIDialectExpression` objects can be parsed using any method you choose, including regular expressions for simple cases or custom AST parsers for complex queries. However, using robust parsers like those integrated into OSSIE converters is recommended to accurately handle nested subqueries, aliases, and dialect-specific syntax that simple string matching might miss.

### Where is metric validation implemented in OSSIE?

Metric validation logic resides in [`validation/validate.py`](https://github.com/apache/ossie/blob/main/validation/validate.py). This module checks that metric names are unique within the semantic model and validates that expressions conform to expected syntax patterns. Running validation before lineage extraction ensures that the SQL strings are well-formed and that references can be successfully resolved to `OSIField` objects without encountering malformed expressions.