How to Implement Custom Data Types Using sdata's sclass System

Implementing custom data types in sdata requires inheriting from sdata.base.Base, implementing to_dict/from_dict methods for serialization, and registering the class with sdata.sclass.register() to enable lazy loading and canonical spec resolution.

The sclass system in the lepy/sdata repository provides a lazy-registry architecture that transforms any Python class into a first-class serializable entity. By implementing custom data types using this system, you gain automatic metadata handling, Super UUID (SUUID) generation, and seamless JSON round-tripping capabilities. This guide walks through the exact steps needed to integrate your custom types into sdata's serialization framework.

Understanding the sclass Architecture

The sdata sclass system centers on a lazy-registry pattern that defers imports until runtime, preventing circular dependency issues while maintaining clean JSON references.

The LazyRegistry Component

At the core of the system is LazyRegistry, implemented in sdata/sclass/lazy_namespace.py (lines 33-75). This class maintains a mapping of short names to fully qualified module specifications (e.g., "MyThing" → "my_pkg.my_mod:MyThing"). The registry dynamically injects __getattr__ and __dir__ into the sdata.sclass package, enabling attribute access to trigger on-demand imports.

When you access sdata.sclass.MyThing, the registry intercepts the call, splits the stored spec into module and class components, imports the module via import_module, and resolves the attribute chain. The resulting class is cached on the package for subsequent accesses, ensuring optimal performance after the initial lazy load.

Core Registration Functions

The public API for extending the registry lives in sdata/sclass/__init__.py (lines 26-33). The register(name, spec) function adds a single entry to the internal _exports dictionary, while register_many handles batch registrations. These functions modify the global _registry instance created during package initialization (lines 7-24).

For serialization workflows, the module provides class_to_spec and spec_to_class functions (lines 43-61). These utilities convert Python classes to canonical "module:QualifiedName" strings and vice versa, enabling sdata to store type information in JSON's "__spec__" field.

Step-by-Step Implementation Guide

Follow these concrete steps to implement a custom data type that integrates fully with sdata's infrastructure.

1. Inherit from the Base Class

All sclasses must inherit from sdata.base.Base (defined in sdata/base.py, lines 28-35). This base class provides essential infrastructure including metadata dictionaries, SUUID handling, and the sdata_class attribute that the registry uses for identification.

# my_pkg/my_mod.py

from sdata.base import Base
import pandas as pd

class MyThing(Base):
    """A custom sclass storing pandas DataFrames with metadata."""
    def __init__(self, df: pd.DataFrame, **kwargs):
        super().__init__(**kwargs)  # Initializes metadata, SUUID, timestamps

        self.data["df"] = df       # Store payload in the data dictionary

Inheriting from Base automatically attaches metadata fields, generates a unique SUUID, and sets up the object for registry operations.

2. Implement Serialization Methods

To support JSON round-tripping via sdata.sclass.to_json and from_json, implement to_dict and from_dict methods. These methods handle the conversion of your domain-specific data to serializable primitives.

    @classmethod
    def from_dict(cls, payload: dict):
        """Reconstruct MyThing from deserialized JSON."""
        df = pd.read_json(payload["df"])
        return cls(df=df, name=payload.get("name", "unnamed"))
    
    def to_dict(self):
        """Convert MyThing to a dictionary for JSON serialization."""
        return {
            "df": self.data["df"].to_json(), 
            "name": self.name
        }

The to_dict method must return JSON-serializable values, while from_dict acts as an alternative constructor that reconstructs the object from the deserialized dictionary.

3. Register Your Custom sclass

During application startup, register your class with the sdata registry to make it available via lazy import and canonical spec resolution.

# my_pkg/__init__.py or application startup code

from sdata.sclass import register

# Map short name to fully qualified spec

register("MyThing", "my_pkg.my_mod:MyThing")

This call adds the entry to LazyRegistry._exports, immediately making the class accessible as sdata.sclass.MyThing without importing my_pkg until the attribute is actually accessed.

4. Use in Production Workflows

Once registered, use your custom type through the sclass namespace or serialize it for storage and transmission.

import pandas as pd
from sdata.sclass import MyThing, to_json, from_json

# Instantiate via the lazy-loaded namespace

df = pd.DataFrame({"x": [1, 2, 3], "y": ["a", "b", "c"]})
thing = MyThing(df=df, name="production_data")

# Serialize includes the canonical spec

payload = to_json(thing, include_spec=True)
print(payload["__spec__"])  # Output: my_pkg.my_mod:MyThing

# Deserialize automatically resolves the spec back to MyThing

restored = from_json(payload)
assert isinstance(restored, MyThing)
assert restored.name == "production_data"

The include_spec=True parameter ensures the JSON contains the "__spec__" field, allowing from_json to reconstruct the exact class type on deserialization using spec_to_class.

Technical Deep Dive

Lazy Loading Mechanism

The registry's _registry.attach() method (called in sdata/sclass/__init__.py) replaces sdata.sclass.__getattr__ with a custom implementation. When you reference sdata.sclass.MyThing, the custom getter checks _exports for the short name, splits the spec string at the colon separator, imports the module portion dynamically, and resolves the attribute chain to retrieve the class.

This design keeps import times minimal because my_pkg is not imported when you import sdata—only when you actually access sdata.sclass.MyThing. The mechanism also prevents circular import issues common in complex data modeling scenarios.

Canonical Spec Resolution

The canonical spec format (module.submodule:ClassName) provides a stable, string-based identifier for classes that persists across process restarts and system boundaries. When from_json encounters a payload, it checks for __spec__ and passes it to spec_to_class, which uses importlib to load the module and getattr chains to resolve nested classes.

If only a short name is present (legacy support), the system falls back to registry resolution via resolve_by_string in lazy_namespace.py, though modern implementations should always write and read the full canonical spec for unambiguous type identification.

Summary

• Inherit from sdata.base.Base to gain metadata, SUUID, and registry compatibility automatically.
• Implement to_dict and from_dict methods to enable JSON serialization of your custom payload data.
• Register using sdata.sclass.register() with a short name and fully qualified module:Class spec to enable lazy loading.
• Reference files correctly: Registry logic lives in sdata/sclass/__init__.py and sdata/sclass/lazy_namespace.py, while sdata/base.py provides the base class.
• Use canonical specs (module:QualifiedName) for stable serialization and deserialization across systems.

Frequently Asked Questions

What is the sclass system in sdata?

The sclass system is a lazy-registry architecture that transforms Python classes into serializable, metadata-rich entities. It enables classes to be referenced by short names, imported on demand, and serialized using canonical specifications. The system lives in the sdata/sclass module and relies on LazyRegistry for dynamic attribute resolution.

Why must custom types inherit from Base?

Inheriting from sdata.base.Base (defined in sdata/base.py) provides essential infrastructure including automatic metadata handling, SUUID generation, timestamp tracking, and the sdata_class attribute required by the registry. Without this base class, objects lack the standardized interface that to_json and from_json expect.

How does the lazy registry prevent circular imports?

The LazyRegistry in sdata/sclass/lazy_namespace.py defers module imports until attribute access occurs. Rather than importing your custom modules when sdata loads, the system waits until you explicitly access sdata.sclass.YourClass. This breaks import chains because sdata never needs to import your application code at startup—your code imports sdata instead.

Can I register multiple classes simultaneously?

Yes. Use the register_many function exported from sdata/sclass/__init__.py (lines 30-33) to register multiple name-to-spec mappings in a single call. This function accepts a dictionary mapping short names to canonical specs and is ideal for registering entire modules of custom types during application initialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}