How AI Berkshire Processes Financial Data: A Complete Technical Guide

AI Berkshire processes financial data through a rigorous six-step pipeline that combines source cross-validation, automated verification via the financial_rigor toolkit, and decimal-exact arithmetic to ensure every metric is mathematically verified before entering investment research reports.

The xbtlin/ai-berkshire repository implements a disciplined, reproducible workflow for financial data validation. Every key metric—from revenue and net profit to market capitalization and valuation ratios—must pass through standardized verification checkpoints defined in policy documents and enforced by pure-Python command-line tools.

The Six-Stage Financial Data Processing Pipeline

Source Selection and Cross-Validation Policy

The foundation of financial data processed by AI Berkshire begins with strict source selection rules defined in skills/financial-data.md. For every data point, the system identifies a primary source (such as macrotrends for US-listed stocks) and a secondary independent source (such as stockanalysis). Original filings from SEC EDGAR, HKEX, or annual report PDFs serve as the ultimate fallback authority.

This dual-source requirement ensures that no single data provider can introduce undetected errors into the research workflow. The policy explicitly mandates that original filings always take precedence over aggregated web data when discrepancies exceed defined thresholds.

Data Extraction and Storage

Raw financial numbers are fetched via scripts or manual lookup from the chosen web portals and stored as CSV files in the data/ directory. While helper utilities like tools/ashare_data.py assist in retrieving A-share data from 东方财富 and 巨潮资讯, the core validation system requires no external libraries. This zero-dependency approach ensures the verification logic remains stable and auditable regardless of third-party package updates.

Automated Cross-Validation Thresholds

Once two independent values are obtained, the system applies strict deviation thresholds:

• ≤1% deviation: Value is automatically accepted
• 1%–5% deviation: Triggers a warning flag (⚠️) requiring analyst review
• >5% deviation: Forces mandatory manual verification against original filings

Special handling rules apply for edge cases. Unlisted companies with only one available source are marked with [估计] to indicate estimation status. Quarterly numbers are automatically overridden by annual data when available, prioritizing audited fiscal year reports over interim statements.

Formal Verification via financial_rigor

The tools/financial_rigor.py command-line interface performs mathematical verification through five distinct subcommands:

1. verify-market-cap: Confirms that price × shares outstanding equals reported market capitalization
2. verify-valuation: Calculates PE, PB, P/FCF, and dividend yield using exact decimal arithmetic
3. cross-validate: Compares multiple source values and returns a consensus median
4. benford: Applies Benford's Law distribution analysis to detect potential data fabrication
5. calc: Performs precise arithmetic calculations without floating-point errors

These checks are invoked automatically by Claude-Code skills at critical validation checkpoints during the research workflow.

Reporting and Exception Handling

Verification results embed directly into generated Markdown research reports with clear visual indicators: ✅ (passed), ⚠️ (warning), or ❌ (failed). When sources disagree within acceptable tolerances, the system reports a "consensus" value representing the statistical median. The tools/report_audit.py utility performs final quality assurance, ensuring every numerical claim in the report carries the corresponding validation flag from financial_rigor.

Core Architecture: Policy Separation from Implementation

AI Berkshire employs a two-layer architecture that separates human-readable policy from executable code:

Skill Layer (skills/*.md): These Markdown files contain the canonical specifications for data sources, formatting standards, and error thresholds. Analysts and AI agents read these documents to understand which web portals to query and how to structure raw data.

Tool Layer (tools/financial_rigor.py): This zero-dependency Python module implements the mathematical verification logic. It accepts raw numbers as command-line arguments, performs decimal-exact arithmetic using Python's Decimal class, and outputs structured audit trails.

This separation allows the system to evolve its data-source list without modifying code, while maintaining mathematical guarantees that every numeric claim is verified and source-checked.

Practical Usage Examples

Market Capitalization Verification

Validate that calculated market cap matches reported values:

python tools/financial_rigor.py verify-market-cap \
    --price 510 \
    --shares 9.11e9 \
    --reported 4.65e12 \
    --currency HKD

Valuation Ratio Calculation

Compute precise valuation metrics from raw inputs:

python tools/financial_rigor.py verify-valuation \
    --price 510 \
    --eps 23.5 \
    --bvps 120 \
    --fcf-per-share 18 \
    --dividend 2.4

Cross-Source Validation

Compare multiple data providers and identify outliers:

python tools/financial_rigor.py cross-validate \
    --field revenue \
    --values '{"年报": 7518, "Yahoo": 7500, "StockAnalysis": 7520}' \
    --unit 亿

Benford's Law Sanity Check

Detect anomalous data patterns that may indicate fabrication:

python tools/financial_rigor.py benford \
    --values '[1234, 2345, 3456, 4567, 5678]'

Exact Arithmetic Calculator

Perform precise calculations without floating-point errors:

python tools/financial_rigor.py calc \
    --expr '510 * 9.11e9'

Summary

• Financial data processed by AI Berkshire follows a mandatory six-step pipeline from source selection to final report auditing.
• The skills/financial-data.md file defines cross-validation policies requiring primary and secondary sources for every metric.
• The tools/financial_rigor.py toolkit provides zero-dependency, decimal-exact verification of market caps, valuation ratios, and source consensus.
• Deviation thresholds of 1% (accept), 1-5% (warning), and >5% (manual check) ensure data quality without unnecessary bottlenecks.
• Original filings from SEC EDGAR and HKEX serve as the authority of last resort when web sources disagree.
• The architecture separates policy (Markdown skills) from implementation (Python tools), enabling rapid updates to data sources while maintaining mathematical rigor.

Frequently Asked Questions

What deviation thresholds trigger manual review in AI Berkshire?

Values with absolute deviations between 1% and 5% trigger a warning (⚠️) requiring analyst attention, while deviations exceeding 5% force mandatory manual verification against original filings such as SEC EDGAR documents or annual report PDFs. Values within 1% are automatically accepted with a ✅ flag.

How does AI Berkshire handle unlisted companies with limited data sources?

Unlisted companies that only have one available data source are explicitly marked with [估计] (estimated) in research reports to flag the lower confidence level. The system still applies mathematical verification to these values but clearly distinguishes them from dual-source validated metrics.

What makes the financial_rigor toolkit different from standard financial libraries?

Unlike libraries that rely on floating-point arithmetic, tools/financial_rigor.py uses Python's Decimal class for exact calculations and operates with zero external dependencies. It functions as a command-line audit tool rather than a data-fetching library, accepting raw numbers as arguments and outputting structured verification flags for direct inclusion in research reports.

Can the data source policies be updated without modifying code?

Yes. Because AI Berkshire separates policy from implementation, data source lists and validation thresholds are defined in human-readable Markdown files within the skills/ directory. The scripts/sync-codex-skills.py utility synchronizes these policy documents with the AI's operational context, allowing analysts to add new primary or secondary sources by editing skills/financial-data.md without touching the Python verification code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}