# How AI Berkshire Handles Different Financial Formats: Normalization and Validation Explained

> Discover how AI Berkshire normalizes and validates diverse financial data formats from multiple sources into a unified, audit-ready schema for accurate analysis.

- Repository: [Xbt Lin/ai-berkshire](https://github.com/xbtlin/ai-berkshire)
- Tags: how-to-guide
- Published: 2026-06-28

---

**AI Berkshire ingests heterogeneous financial data from sources like Tencent 行情, Morningstar, and Xueqiu, then normalizes it into a consistent, audit-ready schema using exact decimal arithmetic, unit conversion utilities, and automated cross-source validation.**

The xbtlin/ai-berkshire repository implements a robust pipeline for reconciling disparate financial formats across global markets. By standardizing Chinese units like "亿" alongside Western denominations and USD valuations, the system ensures reproducible analysis regardless of data origin.

## The Three-Stage Financial Data Pipeline

AI Berkshire processes financial information through a structured workflow that transforms raw inputs into validated, unified metrics.

### Raw Data Ingestion from Multiple Sources

Each data connector fetches information in its native format without premature transformation.

- [`tools/ashare_data.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/ashare_data.py) retrieves real-time quotes and multi-year fundamentals from Tencent 行情 and 东方财富, returning JSON objects with Chinese-specific fields using units such as "亿" (hundred-million) and "万" (ten-thousand).
- [`tools/morningstar_fair_value.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/morningstar_fair_value.py) interfaces with the Morningstar API to capture valuation data in USD, parsing JSON payloads where monetary values follow Western conventions.
- [`tools/xueqiu_scraper.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/xueqiu_scraper.py) extracts numeric fields from Xueqiu timeline pages, processing JSON responses embedded in the scraped HTML.

### Standardization and Unit Conversion

A dedicated utility layer in [`tools/financial_rigor.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/financial_rigor.py) converts heterogeneous representations into a unified internal schema through several specialized mechanisms.

**Exact decimal arithmetic** eliminates floating-point drift by wrapping all numeric values with `Decimal` via `financial_rigor.exact`. This ensures calculations remain reproducible across different Python environments and processor architectures.

**Currency handling** functions like `verify_market_cap` accept a `currency` argument (e.g., "HKD", "RMB", "USD"), allowing the same calculation logic to process valuations across different monetary systems. The currency label propagates through output tables to maintain audit trails.

**Human-readable formatting** utilities include `financial_rigor.fmt_number` for collapsing large integers into "T/B/M" suffixes or Chinese "亿/万亿" units. Helper functions `_fmt_yi` and `_fmt_pct` in [`tools/ashare_data.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/ashare_data.py) automatically scale values exceeding 1e8 to the "亿" unit while formatting percentages consistently.

### Cross-Source Validation

After normalization, AI Berkshire performs rigorous consistency checks to surface discrepancies between data providers.

The `financial_rigor.cross_validate` function accepts a dictionary of `{source: value}` pairs, computes a median consensus, and flags deviations exceeding a configurable tolerance (default 2%). This catches unit mismatches such as HKD 亿 versus RMB 万亿 that might otherwise corrupt analysis.

For anomaly detection regardless of original format, `financial_rigor.benford_check` applies Benford's Law analysis to detect potential data manipulation or transcription errors in the normalized datasets.

## Practical Implementation Examples

### Validating Market Cap with Chinese Units

When processing A-share data that reports market capitalization in "亿" (hundred-million) units:

```python
from tools import financial_rigor as fr

# Price in HKD, shares outstanding, reported market cap

fr.verify_market_cap(
    price=510,
    shares=9.11e9,
    reported_cap=4.65e12,
    currency="HKD"
)

```

This function calculates the theoretical market cap from price and shares, compares it against the reported value, and prints a warning if the deviation exceeds 5%.

### Cross-Validating Revenue Across Different Formats

To reconcile revenue figures where sources use different unit conventions:

```python
from tools import financial_rigor as fr

source_vals = {
    "年报": 7518,          # Unit: 亿 (hundred-million)

    "Yahoo": 7520,        # Unit: 亿

    "EastMoney": 7.5e12   # Unit: 元 (raw)

}

# Normalize EastMoney to 亿 before validation

source_vals["EastMoney"] = source_vals["EastMoney"] / 1e8

fr.cross_validate(
    field_name="revenue",
    source_values=source_vals,
    unit="亿",
    tolerance_pct=2.0
)

```

The tool outputs a median consensus, individual source deviations, and a final consistency flag indicating whether the dataset passes validation.

### Fetching Normalized A-Share Quotes

For command-line retrieval of standardized Chinese equity data:

```bash
python3 tools/ashare_data.py quote 600519

```

The script automatically applies `_fmt_yi` to display market cap as "亿" and turnover as "万", ensuring human-readable output while preserving exact internal representations.

### Processing Morningstar USD Valuations

When ingesting Western-formatted valuation data:

```python
from tools import morningstar_fair_value as ms, financial_rigor as fr

data = ms.fetch_fair_value(symbol="AAPL")

# Returns: {"price": 185.32, "eps": 5.6, "bvps": 22.1, ...}

fr.verify_valuation(
    price=data["price"],
    eps=data["eps"],
    bvps=data["bvps"]
)

```

All ratio calculations (P/E, P/B, ROE) use exact `Decimal` arithmetic from `financial_rigor.exact`, guaranteeing bit-for-bit reproducible results across runs.

## Summary

- **Multi-source ingestion**: Native connectors for Tencent 行情, 东方财富, Morningstar, and Xueqiu handle JSON, CSV, and plain-text formats without premature transformation.
- **Exact arithmetic**: The `financial_rigor` module uses `Decimal` types exclusively to prevent floating-point errors in financial calculations.
- **Unit flexibility**: Built-in converters handle Chinese units ("亿", "万") and Western suffixes ("T/B/M") with automatic scaling based on magnitude.
- **Validation rigor**: Cross-source validation (`cross_validate`) and Benford's Law analysis (`benford_check`) ensure data integrity across disparate providers.

## Frequently Asked Questions

### How does AI Berkshire handle Chinese financial units like "亿"?

AI Berkshire processes Chinese units through dedicated formatters in [`tools/ashare_data.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/ashare_data.py) such as `_fmt_yi`, which automatically scales values to the "亿" (hundred-million) unit when numbers exceed 1e8. The `financial_rigor` module normalizes these into exact `Decimal` objects while preserving display formatting for output.

### What validation methods ensure data accuracy across sources?

The system employs `financial_rigor.cross_validate` to compute median consensus values from multiple providers and flags deviations exceeding 2%. Additionally, `financial_rigor.benford_check` applies statistical analysis to detect anomalous distributions that might indicate data manipulation or transcription errors.

### Can AI Berkshire process both real-time and historical financial data?

Yes. The [`tools/ashare_data.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/ashare_data.py) module fetches real-time quotes alongside multi-year fundamental data, while [`tools/morningstar_fair_value.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/morningstar_fair_value.py) retrieves current valuation metrics. The normalization pipeline handles both temporal dimensions using the same exact arithmetic and unit conversion logic.

### How does the system prevent floating-point errors in calculations?

All numeric values pass through `financial_rigor.exact`, which converts inputs to Python's `Decimal` type before any mathematical operations. This approach eliminates binary floating-point representation errors, ensuring that calculations involving currency conversions, ratio analyses, and cross-validations remain reproducible and audit-ready.