Ossie Dataset Definition Structure: Complete Guide to Schema Modeling

An Ossie dataset definition consists of seven core properties—name, source, primaryKey, uniqueKeys, description, fields, and customExtensions—defined within the Dataset inner class of OsiModel and serialized to JSON for cross-platform semantic modeling.

An Ossie dataset represents the fundamental logical table abstraction inside the Apache Ossie semantic model. As implemented in apache/ossie, this structure bridges business concepts with physical implementations across diverse backends like Iceberg, Polaris, and Salesforce. Understanding the exact schema definition is essential for building conversion pipelines and maintaining data integrity across federated systems.

Core Dataset Properties

The dataset definition lives in OsiModel.java (lines 76‑105) and encapsulates metadata required to generate backend-specific schemas. Each property maps directly to a JSON field during serialization.

Required Identifiers

The name property serves as the unique table identifier. This string value becomes the canonical reference used throughout conversion handlers and must be unique within the model scope.

Data Integrity Constraints

Two properties enforce uniqueness constraints:

• primaryKey – A List<String> containing ordered column names that constitute the table's primary key
• uniqueKeys – A List<List<String>> containing zero or more column sets that must remain unique together

These constraints propagate to backend-specific DDL generation through the conversion pipeline.

Metadata and Extensibility

Additional descriptive and flexible properties include:

• source – Optional provenance string tracking the originating system or file (e.g., "customers__dll")
• description – Human-readable documentation for the dataset
• customExtensions – A List<CustomExtension> providing key-value pairs for vendor-specific attributes and future-proofing

Field Definitions

Each dataset contains a fields property—a List<Field> where individual columns are defined. The Field class (lines 107‑124 in OsiModel.java) specifies four critical attributes.

Column Structure

Every field requires:

• name – The column identifier used in queries and mappings
• description – Optional documentation string for business context
• isTime – Boolean flag indicating whether the column holds temporal data, used for time-series optimizations and partitioning hints

Dialect Expressions

The expressions property contains a List<DialectExpression> objects. Each expression pairs a dialect name (e.g., "iceberg", "sql") with a dialect-specific type definition or expression string.

This polymorphic approach allows a single logical field to render as int for Iceberg while simultaneously mapping to VARCHAR(255) for SQL backends.

Implementation in Source Code

According to the apache/ossie repository, the data structures are implemented as static inner classes within OsiModel:

Source File	Lines	Content
OsiModel.java	76‑105	Dataset class definition with all seven properties
OsiModel.java	107‑124	Field class with name, description, expressions, and time flag

These classes utilize standard Java bean conventions with getters and setters, enabling Jackson serialization to the JSON format consumed by downstream handlers.

Practical Example: Creating a Dataset Programmatically

The following Java code demonstrates constructing a complete dataset definition with multiple fields and dialect expressions:

import org.apache.ossie.converter.polaris.model.OsiModel;
import org.apache.ossie.converter.polaris.model.OsiModel.Dataset;
import org.apache.ossie.converter.polaris.model.OsiModel.Field;
import org.apache.ossie.converter.polaris.model.OsiModel.DialectExpression;
import java.util.List;

OsiModel model = new OsiModel();
Dataset customers = new Dataset();
customers.setName("Customers");
customers.setSource("customers__dll");
customers.setDescription("Customer master data");
customers.setPrimaryKey(List.of("customer_id"));

Field id = new Field();
id.setName("customer_id");
id.setDescription("Unique identifier");
id.setTime(false);
id.setExpressions(List.of(new DialectExpression("iceberg", "int")));

Field email = new Field();
email.setName("email");
email.setDescription("Customer email address");
email.setExpressions(List.of(
    new DialectExpression("iceberg", "string"),
    new DialectExpression("sql", "VARCHAR(255)")
));
email.setTime(false);

customers.setFields(List.of(id, email));
model.setDatasets(List.of(customers));

When serialized, this produces the following JSON structure:

{
  "datasets": [
    {
      "name": "Customers",
      "source": "customers__dll",
      "primaryKey": ["customer_id"],
      "description": "Customer master data",
      "fields": [
        {
          "name": "customer_id",
          "description": "Unique identifier",
          "isTime": false,
          "expressions": [
            { "dialect": "iceberg", "expression": "int" }
          ]
        },
        {
          "name": "email",
          "description": "Customer email address",
          "isTime": false,
          "expressions": [
            { "dialect": "iceberg", "expression": "string" },
            { "dialect": "sql", "expression": "VARCHAR(255)" }
          ]
        }
      ]
    }
  ]
}

Conversion Pipeline Integration

The Dataset and Field definitions serve as the immutable contract passed between conversion handlers. The Ossie conversion pipeline employs specialized handlers to transform these structures:

• DatasetMappingHandler – Processes dataset-level metadata and constraints
• FieldMappingHandler – Translates field definitions into backend-specific column specs
• CustomExtensionHandler – Routes vendor-specific extensions to appropriate transformation logic

These components consume the JSON-serialized dataset definitions to generate Iceberg schemas, SQL DDL, or Salesforce object mappings while preserving the semantic intent captured in the Ossie model.

Summary

• An Ossie dataset definition comprises seven properties: name, source, primaryKey, uniqueKeys, description, fields, and customExtensions.
• Fields contain dialect-agnostic metadata plus a list of DialectExpression objects enabling multi-backend deployment.
• The implementation resides in OsiModel.java lines 76‑124, using inner classes with Jackson-compatible serialization.
• Primary keys and unique keys are specified as string lists and enforced during conversion.
• The structure supports custom extensions for vendor-specific attributes without breaking core schema contracts.

Frequently Asked Questions

How are primary keys defined in an Ossie dataset?

Primary keys are defined using the primaryKey property, which accepts a List<String> of ordered column names. This list determines the column composition and sequence of the primary key constraint, which conversion handlers translate into backend-specific DDL statements such as Iceberg's PRIMARY KEY clause or SQL constraints.

Can a single Ossie field have multiple type definitions?

Yes. The expressions field within a Field object contains a list of DialectExpression objects. Each expression pairs a dialect identifier (like "iceberg" or "sql") with a type-specific string, allowing the same logical column to map to int for Iceberg tables and INTEGER for traditional SQL databases simultaneously.

What is the purpose of customExtensions in a dataset definition?

The customExtensions property provides a List<CustomExtension> for storing vendor-specific or application-specific key-value pairs. This extensibility mechanism allows teams to attach proprietary metadata (such as Salesforce API names or internal data lineage IDs) without modifying the core Ossie schema, ensuring backward compatibility.

Where is the Dataset class located in the Apache Ossie repository?

The Dataset class is defined as a static inner class within OsiModel.java at lines 76‑105, while the nested Field class occupies lines 107‑124. Both classes reside in the org.apache.ossie.converter.polaris.model package and serve as the backbone for the Polaris conversion module.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}