How AI Berkshire Ensures Financial Data Rigor with LLMs: The Financial Rigor Toolkit Explained

AI Berkshire treats every numeric claim generated by LLMs as provisional until programmatically verified through a deterministic pipeline that uses exact-decimal arithmetic, cross-source validation, and statistical sanity checks.

The AI Berkshire repository (xbtlin/ai-berkshire) implements a zero-trust architecture for financial data where large language models produce qualitative insights while a dedicated verification engine handles quantitative validation. This separation ensures that financial data rigor with LLMs is maintained through deterministic recomputation rather than probabilistic generation.

The Four Pillars of Financial Verification

The verification workflow centers on tools/financial_rigor.py, which implements four distinct validation layers to catch computational errors, source discrepancies, and potential data manipulation before publication.

Exact-Decimal Arithmetic

All calculations use Python’s Decimal type to eliminate floating-point drift. The toolkit initializes a shared context _CTX at lines 28-33 in tools/financial_rigor.py with 28-digit precision and ROUND_HALF_EVEN rounding:

from decimal import Decimal, Context, ROUND_HALF_EVEN

_CTX = Context(prec=28, rounding=ROUND_HALF_EVEN)

This context ensures that operations like market-cap multiplication maintain precision across arbitrarily large numbers, preventing the micro-errors that accumulate in standard IEEE 754 floating-point arithmetic.

Market-Cap and Valuation Verification

The verify_market_cap function (lines 61-90) and verify_valuation function (lines 98-160) recompute financial metrics from raw inputs and compare results against LLM-reported figures. The system triggers a caution flag for deviations greater than 1% and a warning for deviations exceeding 5%.

These functions calculate PE ratios, price-to-book (PB), price-to-free-cash-flow (P/FCF), and dividend yields using the exact-decimal context, ensuring that valuation multiples cited in research reports match the underlying price and fundamental data.

Cross-Source Data Validation

The cross_validate function (lines 167-204) aggregates values from multiple independent sources—such as Bloomberg terminals, Yahoo Finance, and company filings—and identifies outliers beyond a configurable tolerance (default 2%). The workflow:

1. Collects the same metric from disparate sources
2. Computes the median as the consensus value
3. Flags any source deviating beyond the threshold

This prevents single-source errors from propagating into final reports, as the LLM prompt itself embeds the source list (e.g., Macrotrends, StockAnalysis, Eastmoney).

Benford-Law Statistical Checks

The benford_check function (lines 211-281) runs a statistical sanity check on sets of financial figures to detect potential data manipulation. By comparing the distribution of leading digits against Benford’s expected logarithmic distribution, the function calculates mean-absolute-deviation (MAD) and chi-square statistics. High deviation scores indicate anomalous data patterns that warrant manual review.

Integration with LLM Workflows

Skills in the skills/ directory invoke the Financial Rigor Toolkit automatically during report generation. For example, skills/investment-research.md (line 57) mandates verification at the data-validation checkpoint:

python3 ~/ai-berkshire/tools/financial_rigor.py verify-market-cap \
    --price {price} --shares {shares} --reported {reported_cap} --currency {currency}

The standard verification workflow follows four steps:

1. Extract raw numbers (price, EPS, shares outstanding) from LLM output using regex patterns
2. Invoke the appropriate sub-command to recompute the metric with exact arithmetic
3. Cross-validate against external sources using cross_validate with a 2% tolerance
4. Record outcomes in the report appendix under the "financial_rigor.py 精确验算" tag

Additionally, tools/report_audit.py performs random sampling on 15% of extracted data points, feeding them back to the LLM for source verification to complement the deterministic checks.

Command-Line Interface for Verification

The toolkit exposes a CLI interface that skills embed directly. Below are the primary verification commands used throughout the AI Berkshire pipeline.

Verify market capitalization calculations:

python3 tools/financial_rigor.py verify-market-cap \
    --price 510 --shares 9.11e9 --reported 4.65e12 --currency HKD

Verify valuation multiples:

python3 tools/financial_rigor.py verify-valuation \
    --price 510 --eps 23.5 --bvps 120 --fcf-per-share 18 --dividend 2.4

Cross-validate revenue figures across sources:

python3 tools/financial_rigor.py cross-validate \
    --field revenue \
    --values '{"年报": 7518, "Yahoo": 7500, "StockAnalysis": 7520}' \
    --unit 亿

Run Benford-law analysis:

python3 tools/financial_rigor.py benford \
    --values '[1234, 2345, 3456, 4567, 5678, 6789, 7890, 8901, 9012]'

Perform exact arithmetic calculations:

python3 tools/financial_rigor.py calc \
    --expr '510 * 9.11e9'

Generate three-scenario valuation models:

python3 tools/financial_rigor.py three-scenario \
    --price 510 --eps 23.5 --shares 9.11 \
    --growth 0.15 0.08 0.00 \
    --pe 25 20 15 \
    --years 3 --currency HKD

Summary

• Exact-decimal arithmetic via Decimal contexts prevents floating-point errors in tools/financial_rigor.py
• Automated verification functions (verify_market_cap, verify_valuation) flag discrepancies >1% as cautions and >5% as warnings
• Cross-source validation aggregates data from multiple financial APIs and uses median consensus with 2% tolerance bands
• Statistical checks via benford_check detect anomalous digit distributions indicative of data manipulation
• Deterministic pipeline ensures every number in AI-generated reports is programmatically verified before publication

Frequently Asked Questions

What is the Financial Rigor Toolkit in AI Berkshire?

The Financial Rigor Toolkit is the core verification engine located at tools/financial_rigor.py that ensures financial data rigor with LLMs by treating all numeric claims as provisional until recomputed and validated. It provides exact-decimal arithmetic, multi-source cross-validation, and statistical sanity checks that run automatically during report generation.

How does AI Berkshire prevent floating-point errors in financial calculations?

The system uses Python’s Decimal type with a custom 28-digit precision context (_CTX) configured for ROUND_HALF_EVEN rounding. This avoids the representation errors inherent in standard floating-point arithmetic when calculating market capitalizations or valuation multiples that span billions of units.

What tolerance levels does AI Berkshire use for data validation?

The toolkit uses a 1% threshold for caution flags and a 5% threshold for warnings when comparing LLM-generated values against recomputed figures. For cross-source validation, the default tolerance is 2%, meaning any source deviating more than 2% from the median consensus value gets flagged for review.

How does the Benford-law check detect manipulated financial data?

The benford_check function analyzes the distribution of leading digits in a dataset of financial figures and compares it against Benford’s expected logarithmic distribution. A high mean-absolute-deviation (MAD) or chi-square statistic indicates that the numbers may have been artificially generated or manipulated, triggering further investigation before report finalization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}