# Schema Evolution in Pathway Data Pipelines: Versioned Metadata and the InnerSchemaField Abstraction

> Pathway tackles schema evolution with versioned metadata and the InnerSchemaField abstraction. Achieve backward-compatible changes in data pipelines without manual migrations.

- Repository: [Pathway/pathway](https://github.com/pathwaycom/pathway)
- Tags: internals
- Published: 2026-03-06

---

**Pathway handles schema evolution through versioned metadata snapshots, Arrow schema-aware column buffers, and the `InnerSchemaField` abstraction, allowing backward-compatible changes without manual migrations.**

Pathway (pathwaycom/pathway) treats **schema evolution** as a first-class concern, isolating schema management from its runtime execution engine. This architecture enables seamless changes—adding, removing, or modifying fields—while guaranteeing that historic data remains accessible and new data follows updated layouts without manual migration steps.

## Versioned Metadata in the Persistence Layer

Pathway’s persistence layer implements **versioned metadata** to track schema changes across time. Every persisted operator snapshot carries a metadata version generated via `MetadataKey::from_components(version, worker_id, rotation_id)` in [`src/persistence/state.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/state.rs). When the schema changes, the system writes a new version while keeping old versions readable.

The engine determines which version to use through the `compute_threshold_time_and_versions` routine. This function selects the *latest stable version* that possesses full metadata for all workers, ensuring the pipeline never operates on a partial schema. The persistence layer can automatically shrink or split batches to align with the new version without breaking existing data streams.

## Schema-Aware Column Buffers

Column buffers in Pathway embed **Arrow schemas** generated from the current `InnerSchemaField` map. The `AppendOnlyColumnBuffer` and `SnapshotColumnBuffer` structures in [`src/persistence/cached_object_storage.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/cached_object_storage.rs) manage these schemas. When schema evolution occurs, the system constructs a new Arrow schema, and the buffer continues writing using the updated layout while preserving earlier rows in their original format.

This design allows the storage layer to maintain multiple physical layouts within the same logical table, with the runtime engine handling the translation between versions during query execution.

## Backward Compatibility via InnerSchemaField

The core of Pathway’s schema flexibility lies in the `InnerSchemaField` abstraction defined in [`src/connectors/data_format.rs`](https://github.com/pathwaycom/pathway/blob/main/src/connectors/data_format.rs). Users define schemas as a `HashMap<String, InnerSchemaField>`, where each field stores a `Type` (including `Optional`, `Any`, `Json`, etc.) and an optional default `Value`.

Adding a new field with a default value or marking it as `Optional` makes the schema **backward-compatible**. When reading older rows that lack the new field, the engine automatically fills missing values with the specified default or `null`, preventing runtime failures.

### Practical Example: Adding a New Field

The following example demonstrates adding a `currency` field to an existing `Order` schema:

```python
import pathway as pw

# Original schema

class Order(pw.Schema):
    order_id: int
    amount: float

orders = pw.io.csv.read("orders_v1/", schema=Order)

# Evolved schema with new field and default value

class OrderV2(pw.Schema):
    order_id: int
    amount: float
    currency: str = "USD"  # Optional with default

# Trigger schema evolution

orders = orders.cast(OrderV2)

```

Behind the scenes, the `cast` call builds a new `HashMap<String, InnerSchemaField>` (see `InnerSchemaField::new` in [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs)). The persistence layer in [`src/persistence/state.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/state.rs) generates a fresh metadata version, while [`src/persistence/cached_object_storage.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/cached_object_storage.rs) handles batch shrinking if needed. The Arrow schema supplied to the column buffers now includes the `currency` field, with existing rows logically inheriting the "USD" default when queried.

## Automatic Compatibility Checks and External Registries

### Connector Validation

When connecting to external systems, Pathway validates table schemas against the supplied definition. In [`src/connectors/sqlite.rs`](https://github.com/pathwaycom/pathway/blob/main/src/connectors/sqlite.rs) and [`src/connectors/postgres.rs`](https://github.com/pathwaycom/pathway/blob/main/src/connectors/postgres.rs), the engine checks for missing columns. If fields are absent but marked as `Optional` or possess default values, the pipeline continues execution; otherwise, it raises a clear error before processing begins.

### Schema Registry Integration

For organizations using centralized schema management, Pathway fetches Avro or Protobuf schemas from **Confluent-compatible Schema Registries**. The `PySchemaRegistrySettings` structure in [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs) handles this integration using the `schema_registry_converter` crate. Fetched schemas convert into `InnerSchemaField` definitions, allowing seamless evolution when the registry version changes, with the same versioned persistence path applying automatically.

## Runtime Re-Evaluation and Default Values

During execution, the Rust engine processes new rows using the *current* schema while older rows retain their original version metadata. When a query accesses a column that did not exist in older schema versions, the engine automatically substitutes the column’s default value (or `null` for `Optional` types) without requiring data migration. This logic operates within the `engine::timestamp` and `engine::dataflow::persist` modules, ensuring consistent results across heterogeneous data versions.

## Summary

- **Versioned snapshots**: Every operator state carries a `MetadataKey` version in [`src/persistence/state.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/state.rs), enabling time-travel reads and safe updates.
- **Arrow-backed buffers**: `AppendOnlyColumnBuffer` and `SnapshotColumnBuffer` in [`src/persistence/cached_object_storage.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/cached_object_storage.rs) separate physical storage layout from logical schema.
- **Flexible field definitions**: The `InnerSchemaField` abstraction in [`src/connectors/data_format.rs`](https://github.com/pathwaycom/pathway/blob/main/src/connectors/data_format.rs) supports defaults and `Optional` types for backward compatibility.
- **Registry integration**: [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs) provides `PySchemaRegistrySettings` for external Confluent Schema Registry support.
- **Runtime resolution**: The engine resolves missing columns using defaults or nulls, eliminating the need for ETL reprocessing when schemas evolve.

## Frequently Asked Questions

### How does Pathway handle missing columns when reading old data with a new schema?

When a query accesses historical data that lacks columns defined in the current schema, Pathway resolves the missing values using the **default value** specified in the `InnerSchemaField` definition, or `null` if the type is `Optional`. This behavior is implemented in the runtime engine and prevents crashes during schema evolution.

### Can Pathway integrate with Confluent Schema Registry for Avro schemas?

Yes. Pathway supports Confluent-compatible Schema Registries through the `PySchemaRegistrySettings` class in [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs), which utilizes the `schema_registry_converter` crate to fetch and convert Avro or Protobuf schemas into internal `InnerSchemaField` definitions.

### What happens to existing data when I add a required field without a default?

If you add a new field without a default value and do not mark it as `Optional`, Pathway’s connector validation (in [`src/connectors/sqlite.rs`](https://github.com/pathwaycom/pathway/blob/main/src/connectors/sqlite.rs) and [`src/connectors/postgres.rs`](https://github.com/pathwaycom/pathway/blob/main/src/connectors/postgres.rs)) will raise an error when encountering existing rows that lack the field. For persisted internal state, the pipeline would fail to resolve the missing value, requiring you to provide a default or mark the field as `Optional` to maintain backward compatibility.

### Where is schema version metadata stored in Pathway?

Schema version metadata is stored in the **persistence layer** via the `MetadataKey` structure in [`src/persistence/state.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/state.rs). Each operator snapshot embeds a version number created by `MetadataKey::from_components(version, worker_id, rotation_id)`, allowing the system to track which schema version corresponds to each data batch.