How AI Berkshire Handles Different Financial Formats: Normalization and Validation Explained

AI Berkshire ingests heterogeneous financial data from sources like Tencent 行情, Morningstar, and Xueqiu, then normalizes it into a consistent, audit-ready schema using exact decimal arithmetic, unit conversion utilities, and automated cross-source validation.

The xbtlin/ai-berkshire repository implements a robust pipeline for reconciling disparate financial formats across global markets. By standardizing Chinese units like "亿" alongside Western denominations and USD valuations, the system ensures reproducible analysis regardless of data origin.

The Three-Stage Financial Data Pipeline

AI Berkshire processes financial information through a structured workflow that transforms raw inputs into validated, unified metrics.

Raw Data Ingestion from Multiple Sources

Each data connector fetches information in its native format without premature transformation.

• tools/ashare_data.py retrieves real-time quotes and multi-year fundamentals from Tencent 行情 and 东方财富, returning JSON objects with Chinese-specific fields using units such as "亿" (hundred-million) and "万" (ten-thousand).
• tools/morningstar_fair_value.py interfaces with the Morningstar API to capture valuation data in USD, parsing JSON payloads where monetary values follow Western conventions.
• tools/xueqiu_scraper.py extracts numeric fields from Xueqiu timeline pages, processing JSON responses embedded in the scraped HTML.

Standardization and Unit Conversion

A dedicated utility layer in tools/financial_rigor.py converts heterogeneous representations into a unified internal schema through several specialized mechanisms.

Exact decimal arithmetic eliminates floating-point drift by wrapping all numeric values with Decimal via financial_rigor.exact. This ensures calculations remain reproducible across different Python environments and processor architectures.

Currency handling functions like verify_market_cap accept a currency argument (e.g., "HKD", "RMB", "USD"), allowing the same calculation logic to process valuations across different monetary systems. The currency label propagates through output tables to maintain audit trails.

Human-readable formatting utilities include financial_rigor.fmt_number for collapsing large integers into "T/B/M" suffixes or Chinese "亿/万亿" units. Helper functions _fmt_yi and _fmt_pct in tools/ashare_data.py automatically scale values exceeding 1e8 to the "亿" unit while formatting percentages consistently.

Cross-Source Validation

After normalization, AI Berkshire performs rigorous consistency checks to surface discrepancies between data providers.

The financial_rigor.cross_validate function accepts a dictionary of {source: value} pairs, computes a median consensus, and flags deviations exceeding a configurable tolerance (default 2%). This catches unit mismatches such as HKD 亿 versus RMB 万亿 that might otherwise corrupt analysis.

For anomaly detection regardless of original format, financial_rigor.benford_check applies Benford's Law analysis to detect potential data manipulation or transcription errors in the normalized datasets.

Practical Implementation Examples

Validating Market Cap with Chinese Units

When processing A-share data that reports market capitalization in "亿" (hundred-million) units:

from tools import financial_rigor as fr

# Price in HKD, shares outstanding, reported market cap

fr.verify_market_cap(
    price=510,
    shares=9.11e9,
    reported_cap=4.65e12,
    currency="HKD"
)

This function calculates the theoretical market cap from price and shares, compares it against the reported value, and prints a warning if the deviation exceeds 5%.

Cross-Validating Revenue Across Different Formats

To reconcile revenue figures where sources use different unit conventions:

from tools import financial_rigor as fr

source_vals = {
    "年报": 7518,          # Unit: 亿 (hundred-million)

    "Yahoo": 7520,        # Unit: 亿

    "EastMoney": 7.5e12   # Unit: 元 (raw)

}

# Normalize EastMoney to 亿 before validation

source_vals["EastMoney"] = source_vals["EastMoney"] / 1e8

fr.cross_validate(
    field_name="revenue",
    source_values=source_vals,
    unit="亿",
    tolerance_pct=2.0
)

The tool outputs a median consensus, individual source deviations, and a final consistency flag indicating whether the dataset passes validation.

Fetching Normalized A-Share Quotes

For command-line retrieval of standardized Chinese equity data:

python3 tools/ashare_data.py quote 600519

The script automatically applies _fmt_yi to display market cap as "亿" and turnover as "万", ensuring human-readable output while preserving exact internal representations.

Processing Morningstar USD Valuations

When ingesting Western-formatted valuation data:

from tools import morningstar_fair_value as ms, financial_rigor as fr

data = ms.fetch_fair_value(symbol="AAPL")

# Returns: {"price": 185.32, "eps": 5.6, "bvps": 22.1, ...}

fr.verify_valuation(
    price=data["price"],
    eps=data["eps"],
    bvps=data["bvps"]
)

All ratio calculations (P/E, P/B, ROE) use exact Decimal arithmetic from financial_rigor.exact, guaranteeing bit-for-bit reproducible results across runs.

Summary

• Multi-source ingestion: Native connectors for Tencent 行情, 东方财富, Morningstar, and Xueqiu handle JSON, CSV, and plain-text formats without premature transformation.
• Exact arithmetic: The financial_rigor module uses Decimal types exclusively to prevent floating-point errors in financial calculations.
• Unit flexibility: Built-in converters handle Chinese units ("亿", "万") and Western suffixes ("T/B/M") with automatic scaling based on magnitude.
• Validation rigor: Cross-source validation (cross_validate) and Benford's Law analysis (benford_check) ensure data integrity across disparate providers.

Frequently Asked Questions

How does AI Berkshire handle Chinese financial units like "亿"?

AI Berkshire processes Chinese units through dedicated formatters in tools/ashare_data.py such as _fmt_yi, which automatically scales values to the "亿" (hundred-million) unit when numbers exceed 1e8. The financial_rigor module normalizes these into exact Decimal objects while preserving display formatting for output.

What validation methods ensure data accuracy across sources?

The system employs financial_rigor.cross_validate to compute median consensus values from multiple providers and flags deviations exceeding 2%. Additionally, financial_rigor.benford_check applies statistical analysis to detect anomalous distributions that might indicate data manipulation or transcription errors.

Can AI Berkshire process both real-time and historical financial data?

Yes. The tools/ashare_data.py module fetches real-time quotes alongside multi-year fundamental data, while tools/morningstar_fair_value.py retrieves current valuation metrics. The normalization pipeline handles both temporal dimensions using the same exact arithmetic and unit conversion logic.

How does the system prevent floating-point errors in calculations?

All numeric values pass through financial_rigor.exact, which converts inputs to Python's Decimal type before any mathematical operations. This approach eliminates binary floating-point representation errors, ensuring that calculations involving currency conversions, ratio analyses, and cross-validations remain reproducible and audit-ready.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}