How to Handle Data Format Differences Between Providers Using OpenBB Standard Models

OpenBB Platform handles data format differences by splitting every provider model into standard fields (common across all sources) and extra fields (provider-specific), automatically merging them via the ProviderInterface class to expose a unified Pydantic schema.

The OpenBB Platform (repository: OpenBB-finance/OpenBB) normalizes heterogeneous financial data from third-party providers like Yahoo Finance, Alpha Vantage, and Tiingo through a sophisticated model abstraction layer. When you handle data format differences between providers using OpenBB standard models, you work with a system that programmatically separates canonical fields from vendor-specific extensions while maintaining strict type safety.

Understanding the Standard vs. Extra Model Architecture

OpenBB resolves provider heterogeneity by enforcing a strict separation between standard models and extra models. Standard models contain fields that all providers agree upon—such as date, open, close, and volume for equity historical data. Extra models contain provider-specific fields like dividendYield from Yahoo Finance or marketCap from Alpha Vantage.

This architecture lives in openbb_platform/core/openbb_core/app/provider_interface.py. The ProviderInterface class orchestrates the entire normalization process by walking the provider registry, extracting raw QueryParams and Data definitions, and constructing concrete Pydantic dataclasses for each model.

How ProviderInterface Merges Provider Schemas

The merging process occurs in four distinct phases during framework initialization:

Extracting Raw Fields from the Registry

The ProviderInterface.__init__ method (lines 98-105) reads the raw QueryParams and Data objects from the registry map via self._registry_map.standard_extra. This collects every field definition across all installed providers for a given endpoint.

Separating Standard and Extra Definitions

Two private methods handle the bifurcation:

• _extract_params (lines 351-409): Separates query parameters into standard and extra buckets
• _extract_data (lines 424-490): Performs the same separation for response data fields

Fields belonging to the "openbb" provider (the canonical reference implementation) become standard; everything else is classified as extra.

Merging Duplicate Field Definitions

When multiple providers define the same field name with different descriptions, types, or constraints, the _merge_fields method (lines 174-239) reconciles them. The merge strategy keeps the most informative description and unions the type hints to ensure compatibility across all sources.

Generating Unified Dataclasses

The interface generates four distinct dataclasses for each model:

• StandardParams and ExtraParams for query parameters (via make_dataclass in _generate_params_dc)
• StandardData and ExtraData for response bodies (via create_model in _generate_data_dc)

Finally, _generate_return_schema (lines 613-648) combines these into a single merged schema that FastAPI injects as a unified model containing both standard and extra fields.

Implementing Provider-Specific Models

To add a new provider with custom fields, define your model by inheriting from the base classes. The ProviderInterface automatically detects and integrates these extensions.

# openbb_platform/providers/custom/openbb_custom/models/custom_equity.py

from openbb_core.provider.abstract.fetcher import Fetcher, Data, QueryParams
from pydantic import Field

class CustomEquityQueryParams(QueryParams):
    # Inherit the canonical query params (symbol, start_date, etc.)

    # and add a provider-specific optional flag.

    include_dividends: bool = Field(default=False, description="Include dividend data.")

class CustomEquityData(Data):
    # Standard fields are pulled from the OpenBB core already.

    # Add a provider-specific field.

    dividend_yield: float | None = Field(
        default=None,
        description="Dividend yield reported by the custom provider."
    )

When the OpenBB registry discovers this module, ProviderInterface will:

• Add include_dividends to the extra query params for the model
• Add dividend_yield to the extra data model
• Merge both sets so the endpoint returns a single CustomEquity schema containing standard fields (e.g., open, close) and dividend_yield

Accessing Unified Models in Application Code

You can interact with the generated models programmatically through the ProviderInterface instance:

from openbb_core.provider import ProviderInterface

# Obtain the generated dataclasses for the "EquityHistorical" model

pi = ProviderInterface()
StandardParams = pi.params["EquityHistorical"]["standard"]
ExtraParams    = pi.params["EquityHistorical"]["extra"]
StandardData   = pi.data["EquityHistorical"]["standard"]
ExtraData      = pi.data["EquityHistorical"]["extra"]

# Build a query that uses the extra flag for the custom provider

query = StandardParams(symbol="AAPL", start_date="2024-01-01")
extra  = ExtraParams(include_dividends=True)

# Call the router (FastAPI takes care of dependency injection)

results = pi.create_executor().fetch("EquityHistorical", query, extra)

# `results` is a Pydantic model containing both standard and extra attributes

print(results.dividend_yield)   # May be None if the provider didn't supply it

Inspecting the Merged Schema

For documentation or debugging purposes, extract the complete JSON schema:

from openbb_core.provider import ProviderInterface

pi = ProviderInterface()
merged_model = pi.return_schema["EquityHistorical"]
print(merged_model.schema_json(indent=2))

The output lists all fields with their combined descriptions, types, and json_schema_extra metadata encoding provider-specific constraints.

Key Source Files and Implementation Details

File	Role
openbb_platform/core/openbb_core/app/provider_interface.py	Core logic for extracting, merging, and building standard/extra models (ProviderInterface class, _merge_fields, _extract_params, _extract_data)
openbb_core/provider/standard_models/*.py	Canonical field definitions used as the "openbb" reference model
openbb_platform/providers/*/models/*.py	Provider-specific model implementations that add or override fields
openbb_platform/core/tests/provider/standard_models/test_standard_models.py	Test suite ensuring standard models contain expected FieldInfo objects
openbb_platform/core/tests/provider/abstract/test_query_params.py	Tests for merging behavior across provider query parameters

Summary

• Standard models contain canonical fields shared across all providers, while extra models contain vendor-specific extensions
• The ProviderInterface class automatically extracts, separates, and merges field definitions from openbb_platform/core/openbb_core/app/provider_interface.py
• Duplicate fields are merged via _merge_fields, keeping the most descriptive metadata and unioning type hints
• New providers add fields by inheriting from QueryParams and Data base classes without modifying core code
• The system exposes a unified schema via return_schema that FastAPI uses for request validation and response serialization

Frequently Asked Questions

What is the difference between standard and extra models in OpenBB?

Standard models contain fields defined by the canonical "openbb" provider that all data sources must implement (e.g., open, high, low, close for prices). Extra models contain provider-specific fields (e.g., adjusted_close, dividend_amount) that only certain sources provide. The ProviderInterface class maintains these as separate Pydantic dataclasses but merges them for the final API schema.

How does OpenBB merge conflicting field definitions from different providers?

When two providers define the same field name differently, the _merge_fields method in provider_interface.py (lines 174-239) reconciles the conflict by preserving the most informative description and creating a union of the type annotations. This ensures the resulting model accepts data from either provider while maintaining strict validation.

Can I add custom fields to a provider without modifying core OpenBB code?

Yes. Create a new provider module in openbb_platform/providers/ and define classes inheriting from QueryParams and Data. Add your custom fields using Pydantic's Field() definitions. The ProviderInterface automatically detects these during registry initialization and adds them to the extra models without requiring changes to the core platform code.

Where are the canonical standard model definitions stored?

The canonical definitions reside in openbb_core/provider/standard_models/*.py. These files establish the baseline fields that constitute the "standard" portion of every model. When the ProviderInterface initializes, it treats fields defined in these files as the reference standard, while all other provider implementations contribute to the extra models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}