# How Financial Data Privacy Is Handled in ai-berkshire: Public Data and Cross-Validation

> Discover how ai-berkshire protects financial data privacy by using public data and cross-validation from independent sources, ensuring security and transparency.

- Repository: [Xbt Lin/ai-berkshire](https://github.com/xbtlin/ai-berkshire)
- Tags: how-to-guide
- Published: 2026-07-04

---

**ai-berkshire treats financial data as publicly sourced information, requiring cross-validation from two independent open-access providers while strictly avoiding proprietary datasets, credentials, or private API endpoints.**

The ai-berkshire repository implements a rigorous **financial data privacy** framework that relies exclusively on public sources rather than confidential feeds. According to the project’s security posture and skill documentation, the system never processes non-public datasets, ensuring complete transparency while maintaining analytical rigor through mandatory dual-source verification.

## Open-Access Data Provider Requirements

The repository restricts all data ingestion to **publicly available sources** only. As defined in [`skills/financial-data.md`](https://github.com/xbtlin/ai-berkshire/blob/main/skills/financial-data.md), acceptable providers include Macrotrends, StockAnalysis, SEC EDGAR, Aastocks, Eastmoney, and cninfo. This policy eliminates any requirement for:

- Private API keys or authentication tokens
- Subscription-based proprietary databases
- Internal corporate financial systems

The documentation explicitly states that every key metric must originate from verifiable public channels, ensuring that **no confidential information** ever enters the processing pipeline.

## Dual-Source Verification Protocol

Before any financial figure is accepted, the system retrieves the metric **from two independent sources**. This cross-validation occurs within the validation utilities, where the error rate is calculated using the formula:

```

|source 1 – source 2| ÷ source 1 × 100%

```

The implementation relies on helper functions in [`tools/financial_rigor.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/financial_rigor.py) to perform exact arithmetic comparisons, while [`tools/report_audit.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/report_audit.py) audits generated reports to ensure proper citation and handling standards.

### Error-Rate Thresholds

The validation workflow applies strict thresholds to determine data reliability:

- **Error ≤ 1%**: Data is accepted and marked as validated
- **Error 1–5%**: Flagged for minor discrepancy requiring review
- **Error > 5%**: Rejected and escalated to original filing verification

When discrepancies arise between aggregated data sources and original regulatory filings (10-K, annual report PDFs, or HKEX disclosures), the **original filing takes precedence** according to the source hierarchy defined in the skill documentation.

## Handling Non-Listed Companies and Estimates

For **non-listed companies** where only a single public source exists, the repository applies a distinct privacy-preserving pattern. Rather than fabricating or inferring additional sources, the data is explicitly marked as an estimate using the tag `[估计]` (estimated). 

This approach maintains transparency without compromising the dual-source principle for publicly traded entities. As documented in the Financial-Data Skill:

> *“本规范适用于所有涉及企业财务数据的研究。每个关键数据必须来自两个独立来源，误差 > 1% 须标记。”* — See the “数据源优先级” and “执行规范” sections in [`skills/financial-data.md`](https://github.com/xbtlin/ai-berkshire/blob/main/skills/financial-data.md).

## Security Policy and Credential Management

The repository’s [`SECURITY.md`](https://github.com/xbtlin/ai-berkshire/blob/main/SECURITY.md) file enforces a strict **no-secrets policy** that explicitly prohibits:

- Storing API credentials or access tokens
- Implementing private endpoint connections
- Processing personally identifiable financial information

This security posture ensures that the entire workflow operates without authentication barriers, making the codebase fully reproducible and auditable by third parties.

## Implementation Examples

The following Python patterns demonstrate the privacy-preserving validation logic without requiring private credentials.

### Validating Revenue from Dual Sources

```python
import requests
import json

def fetch_macrotrends(ticker: str) -> float:
    url = f"https://api.macrotrends.net/stocks/charts/{ticker}"
    data = requests.get(url).json()
    return float(data["revenue"])

def fetch_stockanalysis(ticker: str) -> float:
    url = f"https://api.stockanalysis.com/stocks/{ticker}/financials"
    data = requests.get(url).json()
    return float(data["revenue"])

def compare_values(val1: float, val2: float) -> float:
    return abs(val1 - val2) / val1 * 100

ticker = "PDD"
rev1 = fetch_macrotrends(ticker)
rev2 = fetch_stockanalysis(ticker)
error_pct = compare_values(rev1, rev2)

if error_pct <= 1:
    print(f"✅ Revenue {rev1:,} (error {error_pct:.2f}%) – sources validated")
elif error_pct <= 5:
    print(f"⚠️ Minor discrepancy ({error_pct:.2f}%) – review reasons")
else:
    print(f"❌ Major discrepancy – consult original filing")

```

### Marking Estimates for Single-Source Entities

```python
def estimate_metric(ticker: str, source: str) -> str:
    # Retrieves value from single public report

    value = fetch_from_public_report(ticker, source)
    return f"{value:,} [估计] – only one source available"

print(estimate_metric("米哈游", "cninfo"))

```

These examples mirror the repository’s cross-validation logic while accessing only public endpoints.

## Summary

- **Only public data sources** are utilized, eliminating credential management and private access requirements
- **Dual-source verification** with calculated error rates ensures data integrity before acceptance
- **Original filing precedence** resolves conflicts by prioritizing regulatory submissions over aggregated data
- **Explicit estimation markers** flag non-listed company data when single-source limitations apply
- **No secrets storage** according to [`SECURITY.md`](https://github.com/xbtlin/ai-berkshire/blob/main/SECURITY.md), ensuring complete transparency in data handling

## Frequently Asked Questions

### Does ai-berkshire store or process any private financial credentials?

No. The repository explicitly prohibits storing API keys, authentication tokens, or access credentials in [`SECURITY.md`](https://github.com/xbtlin/ai-berkshire/blob/main/SECURITY.md). All data retrieval operates against public endpoints requiring no authentication, ensuring that no private financial infrastructure is accessed or exposed.

### How does the system handle financial data for non-listed companies?

When analyzing non-listed entities where only one public source exists, the system marks the data as an estimate using the `[估计]` tag. No cross-validation is performed in these cases, but the limitation is transparently disclosed to downstream users rather than hidden or fabricated.

### What happens when two independent data sources disagree significantly?

If the calculated error rate exceeds 5%, the data is rejected pending manual review. When the discrepancy falls between 1% and 5%, the system flags a minor discrepancy. In all cases where aggregated data conflicts with original regulatory filings (10-K, annual reports), the original filing takes precedence according to the hierarchy in [`skills/financial-data.md`](https://github.com/xbtlin/ai-berkshire/blob/main/skills/financial-data.md).

### Where is the data validation logic implemented in the codebase?

The core validation utilities reside in [`tools/financial_rigor.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/financial_rigor.py), which handles exact arithmetic comparisons and error-rate calculations. Report auditing and citation verification occur in [`tools/report_audit.py`](https://github.com/xbtlin/ai-berkshire/blob/main/tools/report_audit.py), while the policy specifications and source hierarchies are documented in [`skills/financial-data.md`](https://github.com/xbtlin/ai-berkshire/blob/main/skills/financial-data.md).