# How sdata's Metadata Attribute System Handles Type Conversion and Validation Errors

> sdata's metadata attribute system uses fail-soft architecture to automatically coerce values, log conversion errors, and optionally validate required fields. Learn how sdata handles type conversion and validation errors.

- Repository: [lepy/sdata](https://github.com/lepy/sdata)
- Tags: deep-dive
- Published: 2026-03-05

---

**sdata's metadata attribute system implements a fail-soft architecture where the `Attribute` class automatically coerces incoming values to specified dtypes, logs conversion failures without raising exceptions, and provides optional validation for required fields.**

The `sdata` library provides a robust metadata container for scientific datasets, implemented primarily in [`sdata/metadata.py`](https://github.com/lepy/sdata/blob/main/sdata/metadata.py). Its metadata attribute system normalizes heterogeneous data inputs through a dedicated `Attribute` class that manages type conversion, missing value handling, and validation error logging. Understanding this internal mechanism helps developers predict how malformed or mismatched data will behave during bulk metadata operations.

## The Attribute Class and Type Determination

The foundation of the system lies in the **`Attribute`** class, which encapsulates individual metadata entries. Each attribute stores its target data type in a `dtype` property and manages value conversion through a centralized setter pipeline.

### Determining Target Data Types

When instantiating an attribute via `Attribute.__init__`, the system first establishes the target type. If the caller provides a `dtype` argument, it is stored directly via `_set_dtype`; otherwise, the system invokes `Attribute.guess_dtype` to infer the type from the initial value. This logic resides in [`sdata/metadata.py`](https://github.com/lepy/sdata/blob/main/sdata/metadata.py) lines 95–104.

### The Conversion Pipeline in _set_value

Actual type casting occurs in the `_set_value` method (lines 28–57). This method looks up the concrete Python type from the **`Attribute.DTYPES`** mapping, then attempts to cast the incoming value. The implementation handles several special cases:

- **Lists**: Strings containing commas are split into Python lists; empty strings become empty lists.
- **Booleans**: Recognizes string representations like `"1"`, `"true"`, `"yes"` as logical `True`.
- **Missing values**: Numeric types receive `np.nan`, while other types receive `None`.
- **Empty strings**: Preserved as `""` for string dtypes.

## Error Handling Strategy

Unlike strict validation frameworks, sdata prioritizes data ingestion continuity over immediate correctness.

### Silent Logging vs. Raising Exceptions

If `_set_value` encounters a value that cannot be coerced to the target dtype, it traps the resulting `ValueError` in an exception block (lines 58–60). Rather than propagating the error to the caller, the system logs a detailed error message using `logger.error()` and leaves the attribute value unchanged—typically as `None` or its previous state. This design allows bulk metadata loading via `Metadata.update_from_dict` to process hundreds of attributes without failing on individual corrupt entries.

### Graceful Fallbacks for Edge Cases

The system implements defensive defaults for ambiguous inputs:

- **Non-list/non-string inputs** that cannot be interpreted as lists raise a `ValueError` (line 40), which is caught and logged.
- **Empty required fields** are permitted during initial storage but flagged later during validation.

## Metadata Container Operations

Individual `Attribute` instances are aggregated within the **`Metadata`** class, which provides high-level batch operations and validation.

### Bulk Loading and Dtype Guessing

The `Metadata.set_attr` method (lines 54–66) and `Metadata.update_from_dict` (lines 13–36) streamline attribute creation. When `guess_dtype=True`, the container first calls `Metadata.guess_dtype_from_value` to infer types before delegating to the `Attribute` constructor. This enables automatic type detection for string representations of numbers, booleans, and dates during dictionary imports.

### Required Field Validation

After population, completeness checking is performed via `Metadata.is_complete()` (lines 94–99). This method iterates through all attributes marked with `required=True` and verifies that stored values are not empty. Unlike type conversion errors, validation failures here are returned as boolean results rather than exceptions, allowing applications to decide whether to proceed with incomplete metadata.

## Practical Code Examples

The following examples demonstrate the conversion and error handling behaviors:

```python
from sdata.metadata import Metadata, Attribute

# Correct type conversion

meta = Metadata()
meta.add("force", "12.5", dtype="float", unit="N")
print(meta.get("force").value, meta.get("force").dtype)

# Output: 12.5 float

# Failed conversion with silent error logging

meta.add("age", "twenty", dtype="int")
print(meta.get("age").value)   # Output: None (conversion failed, error logged)

```

```python

# List parsing from comma-separated strings

meta.add("tags", "red, blue,green", dtype="list")
print(meta.get("tags").value)  

# Output: ['red', 'blue', 'green']

# Required field validation

meta.add("sample_id", "", required=True)
print(meta.is_complete())      # Output: False

```

```python

# Bulk loading with automatic dtype detection

data = {
    "temperature": "23.7",        # guessed as float

    "valid": "True",              # guessed as bool

    "notes": None,                # kept as None

}
meta.update_from_dict(data)
print(meta.get("temperature").value, meta.get("temperature").dtype)  # 23.7 float

print(meta.get("valid").value, meta.get("valid").dtype)              # True bool

```

## Summary

- **Type coercion** occurs in `Attribute._set_value`, which casts inputs to the target dtype defined in `Attribute.DTYPES` or leaves them as `None` on failure.
- **Error suppression** ensures that `ValueError` exceptions during conversion are logged but not raised, preserving pipeline continuity during bulk operations.
- **Special handling** normalizes lists, booleans, and missing values (`np.nan` for numeric types, `None` otherwise).
- **Validation separation** distinguishes between type conversion (automatic) and required field checking (explicit via `Metadata.is_complete()`).
- **Source locations**: Core logic resides in [`sdata/metadata.py`](https://github.com/lepy/sdata/blob/main/sdata/metadata.py), specifically lines 28–60 for conversion and lines 94–99 for validation.

## Frequently Asked Questions

### What happens when sdata cannot convert a metadata value to the specified dtype?

The conversion error is caught internally in `Attribute._set_value` (lines 58–60), logged as an error message, and the attribute retains its previous value or defaults to `None`. No exception propagates to the calling code, allowing batch imports to continue processing remaining fields.

### How does sdata handle missing or null values during type conversion?

Numeric attributes receive `numpy.nan` when values are missing, while non-numeric types receive Python `None`. Empty strings are preserved for string dtypes but become empty lists for list dtypes, ensuring downstream code receives type-appropriate null representations.

### Can sdata automatically detect data types during bulk metadata import?

Yes. When using `Metadata.update_from_dict()` with `guess_dtype=True`, the system analyzes each value to infer the appropriate dtype via `Metadata.guess_dtype_from_value` before constructing the `Attribute`, enabling automatic type detection for strings representing numbers and booleans.

### How do I check if all required metadata attributes have valid values?

Call `Metadata.is_complete()` (lines 94–99 in [`sdata/metadata.py`](https://github.com/lepy/sdata/blob/main/sdata/metadata.py)). This method returns `False` if any attribute marked with `required=True` contains an empty value, and `True` otherwise. Note that this check occurs after conversion, so failed conversions resulting in `None` will also trigger incompleteness for required fields.