How to Handle Column-Level Lineage for Metrics in Ossie

OSSIE derives column-level lineage for metrics by parsing the SQL expressions stored in OSIDialectExpression objects and mapping referenced columns back to their originating OSIField definitions within the semantic model.

Apache OSSIE (Open Source Semantic Information Exchange) models metrics as first-class objects that encapsulate dialect-specific SQL expressions, enabling precise tracking of data lineage from calculated metrics back to source dataset columns. By examining these expressions and resolving column references against the semantic model's field definitions, you can build complete column-level lineage graphs for downstream data governance platforms like Snowflake or OrionBelt.

Understanding OSSIE's Metric Architecture

OSSIE represents metrics through a hierarchical model defined in python/src/ossie/models.py. This architecture separates the metric's identity from its implementation across different SQL dialects.

OSIMetric and OSIExpression Structure

The OSIMetric class ([models.py](https://github.com/apache/ossie/blob/main/python/src/ossie/models.py#L39-L48)) serves as the container for metric definitions, holding the metric name and an associated OSIExpression. The OSIExpression class ([models.py](https://github.com/apache/ossie/blob/main/python/src/ossie/models.py#L80-L86)) encapsulates a collection of dialect-specific implementations, allowing the same logical metric to be rendered as Snowflake SQL, ANSI SQL, or MAQL depending on the target platform.

Dialect-Specific Expression Storage

Each dialect implementation is stored as an OSIDialectExpression ([models.py](https://github.com/apache/ossie/blob/main/python/src/ossie/models.py#L71-L78)). This object pairs a dialect identifier (OSIDialect) with the raw expression string. Because OSSIE stores these expressions as strings rather than parsed ASTs, column-level lineage extraction requires external parsing to identify column references within the SQL text.

Implementing Column-Level Lineage Extraction

The typical workflow to obtain column-level lineage involves defining metrics with explicit field references, parsing the SQL to extract identifiers, and mapping those identifiers back to the dataset fields.

Step 1: Define Metrics with Fully-Qualified References

When creating an OSIMetric, use fully-qualified field names (e.g., Orders.amount) in your dialect expressions. This practice ensures that lineage extraction can unambiguously map columns to their source datasets.

from ossie.models import OSIMetric, OSIExpression, OSIDialectExpression, OSIDialect

metric = OSIMetric(
    name="TotalSales",
    expression=OSIExpression(
        dialects=[
            OSIDialectExpression(
                dialect=OSIDialect.SNOWFLAKE,
                expression="SUM(Orders.amount * Orders.price)"
            ),
            OSIDialectExpression(
                dialect=OSIDialect.ANSI_SQL,
                expression="SUM(Orders.amount * Orders.price)"
            ),
        ]
    ),
    description="Total sales amount in monetary units",
)

Step 2: Parse Dialect Expressions to Identify Columns

Because OSSIE does not enforce parsing internally, you must use a SQL parser—such as sqlglot via the Snowflake converter or another dialect-specific parser—to walk the expression AST and locate column references. The converters in converters/snowflake/src/osi_to_snowflake_yaml_converter.py demonstrate how to transform dialect expressions while preserving lineage information.

Step 3: Map Column Identifiers to OSIField Objects

Once you extract column identifiers from the parsed expression, resolve each reference to the corresponding OSIField object in the dataset. This resolution creates the lineage link from the metric back to the source columns.

import sqlglot
from ossie.models import OSIDataset, OSIField, OSIMetric, OSIDialect

def extract_lineage(metric: OSIMetric, datasets: list[OSIDataset]) -> dict[str, list[OSIField]]:
    """Return a mapping of column names ↦ OSIField objects used by the metric."""
    lineage = {}
    # Choose a dialect that we can parse (e.g., SNOWFLAKE / ANSI_SQL)

    expr = next(
        d.expression
        for d in metric.expression.dialects
        if d.dialect in {OSIDialect.SNOWFLAKE, OSIDialect.ANSI_SQL}
    )
    # Parse the expression with sqlglot

    ast = sqlglot.parse_one(expr, read=metric.expression.dialects[0].dialect.lower())
    for col in ast.find_all(sqlglot.exp.Column):
        col_name = f"{col.table}.{col.name}"
        # Locate the field in the supplied datasets

        for ds in datasets:
            if ds.name.lower() == col.table.lower():
                field = next((f for f in ds.fields or [] if f.name == col.name), None)
                if field:
                    lineage.setdefault(col_name, []).append(field)
    return lineage

Step 4: Serialize Lineage for Downstream Tools

After resolving the lineage mappings, serialize the complete semantic model—including datasets, fields, and metrics—to OSSIE-compatible YAML or JSON. This format can be consumed by downstream lineage tools or converted to platform-specific metadata.

from ossie.models import OSIDocument

doc = OSIDocument(
    version="0.2.0.dev0",
    semantic_model=[
        # ... include datasets, relationships, and the metric defined above ...

    ],
)

print(doc.to_osi_yaml())

Validation and Converter Integration

The validation/validate.py module ensures that metric names are unique and that expressions are syntactically valid before lineage extraction begins. This validation step prevents errors during the parsing phase.

Converters such as the Snowflake converter and OrionBelt ([converters/orionbelt/README.md](https://github.com/apache/ossie/blob/main/converters/orionbelt/README.md)) provide concrete implementations for parsing dialect-specific expressions. These converters walk the expression AST to locate column references and emit OSSIE-compatible lineage objects, ensuring that lineage information is retained when translating metrics across vendor boundaries.

Summary

• OSIMetric objects contain OSIExpression instances that store raw SQL strings in OSIDialectExpression containers, enabling multi-dialect support.
• Column-level lineage requires parsing these expression strings using tools like sqlglot or converter-specific parsers to extract column references.
• Extracted column identifiers must be mapped to OSIField objects within the corresponding datasets to establish complete lineage paths.
• The validation/validate.py module ensures metric integrity before lineage extraction, while converters handle dialect-specific parsing and platform translation.
• Lineage metadata can be serialized to YAML/JSON via OSIDocument for integration with external data governance tools.

Frequently Asked Questions

What is the relationship between OSIMetric and OSIExpression?

An OSIMetric represents a single logical metric (such as "Total Revenue") and contains exactly one OSIExpression object. The OSIExpression acts as a container for multiple OSIDialectExpression instances, each representing the metric's implementation in a specific SQL dialect (Snowflake, ANSI SQL, MAQL, etc.). This separation allows the same metric definition to be deployed across different database platforms while maintaining a single source of truth for lineage purposes.

How does OSSIE handle different SQL dialects when extracting lineage?

OSSIE stores dialect-specific SQL as raw strings within OSIDialectExpression objects tagged with an OSIDialect enum value. When extracting lineage, you select the appropriate dialect expression and use a parser compatible with that dialect—such as the Snowflake converter for Snowflake SQL or a generic SQL parser for ANSI SQL. The converters/snowflake/src/osi_to_snowflake_yaml_converter.py implementation demonstrates how to parse these expressions while preserving column references for lineage tracking.

Can lineage be extracted without using external parsers like sqlglot?

While OSSIE does not provide built-in SQL parsing, the raw expression strings in OSIDialectExpression objects can be parsed using any method you choose, including regular expressions for simple cases or custom AST parsers for complex queries. However, using robust parsers like those integrated into OSSIE converters is recommended to accurately handle nested subqueries, aliases, and dialect-specific syntax that simple string matching might miss.

Where is metric validation implemented in OSSIE?

Metric validation logic resides in validation/validate.py. This module checks that metric names are unique within the semantic model and validates that expressions conform to expected syntax patterns. Running validation before lineage extraction ensures that the SQL strings are well-formed and that references can be successfully resolved to OSIField objects without encountering malformed expressions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}