How Financial Data Privacy Is Handled in ai-berkshire: Public Data and Cross-Validation

ai-berkshire treats financial data as publicly sourced information, requiring cross-validation from two independent open-access providers while strictly avoiding proprietary datasets, credentials, or private API endpoints.

The ai-berkshire repository implements a rigorous financial data privacy framework that relies exclusively on public sources rather than confidential feeds. According to the project’s security posture and skill documentation, the system never processes non-public datasets, ensuring complete transparency while maintaining analytical rigor through mandatory dual-source verification.

Open-Access Data Provider Requirements

The repository restricts all data ingestion to publicly available sources only. As defined in skills/financial-data.md, acceptable providers include Macrotrends, StockAnalysis, SEC EDGAR, Aastocks, Eastmoney, and cninfo. This policy eliminates any requirement for:

• Private API keys or authentication tokens
• Subscription-based proprietary databases
• Internal corporate financial systems

The documentation explicitly states that every key metric must originate from verifiable public channels, ensuring that no confidential information ever enters the processing pipeline.

Dual-Source Verification Protocol

Before any financial figure is accepted, the system retrieves the metric from two independent sources. This cross-validation occurs within the validation utilities, where the error rate is calculated using the formula:

|source 1 – source 2| ÷ source 1 × 100%

The implementation relies on helper functions in tools/financial_rigor.py to perform exact arithmetic comparisons, while tools/report_audit.py audits generated reports to ensure proper citation and handling standards.

Error-Rate Thresholds

The validation workflow applies strict thresholds to determine data reliability:

• Error ≤ 1%: Data is accepted and marked as validated
• Error 1–5%: Flagged for minor discrepancy requiring review
• Error > 5%: Rejected and escalated to original filing verification

When discrepancies arise between aggregated data sources and original regulatory filings (10-K, annual report PDFs, or HKEX disclosures), the original filing takes precedence according to the source hierarchy defined in the skill documentation.

Handling Non-Listed Companies and Estimates

For non-listed companies where only a single public source exists, the repository applies a distinct privacy-preserving pattern. Rather than fabricating or inferring additional sources, the data is explicitly marked as an estimate using the tag [估计] (estimated).

This approach maintains transparency without compromising the dual-source principle for publicly traded entities. As documented in the Financial-Data Skill:

“本规范适用于所有涉及企业财务数据的研究。每个关键数据必须来自两个独立来源，误差 > 1% 须标记。” — See the “数据源优先级” and “执行规范” sections in skills/financial-data.md.

Security Policy and Credential Management

The repository’s SECURITY.md file enforces a strict no-secrets policy that explicitly prohibits:

• Storing API credentials or access tokens
• Implementing private endpoint connections
• Processing personally identifiable financial information

This security posture ensures that the entire workflow operates without authentication barriers, making the codebase fully reproducible and auditable by third parties.

Implementation Examples

The following Python patterns demonstrate the privacy-preserving validation logic without requiring private credentials.

Validating Revenue from Dual Sources

import requests
import json

def fetch_macrotrends(ticker: str) -> float:
    url = f"https://api.macrotrends.net/stocks/charts/{ticker}"
    data = requests.get(url).json()
    return float(data["revenue"])

def fetch_stockanalysis(ticker: str) -> float:
    url = f"https://api.stockanalysis.com/stocks/{ticker}/financials"
    data = requests.get(url).json()
    return float(data["revenue"])

def compare_values(val1: float, val2: float) -> float:
    return abs(val1 - val2) / val1 * 100

ticker = "PDD"
rev1 = fetch_macrotrends(ticker)
rev2 = fetch_stockanalysis(ticker)
error_pct = compare_values(rev1, rev2)

if error_pct <= 1:
    print(f"✅ Revenue {rev1:,} (error {error_pct:.2f}%) – sources validated")
elif error_pct <= 5:
    print(f"⚠️ Minor discrepancy ({error_pct:.2f}%) – review reasons")
else:
    print(f"❌ Major discrepancy – consult original filing")

Marking Estimates for Single-Source Entities

def estimate_metric(ticker: str, source: str) -> str:
    # Retrieves value from single public report

    value = fetch_from_public_report(ticker, source)
    return f"{value:,} [估计] – only one source available"

print(estimate_metric("米哈游", "cninfo"))

These examples mirror the repository’s cross-validation logic while accessing only public endpoints.

Summary

• Only public data sources are utilized, eliminating credential management and private access requirements
• Dual-source verification with calculated error rates ensures data integrity before acceptance
• Original filing precedence resolves conflicts by prioritizing regulatory submissions over aggregated data
• Explicit estimation markers flag non-listed company data when single-source limitations apply
• No secrets storage according to SECURITY.md, ensuring complete transparency in data handling

Frequently Asked Questions

Does ai-berkshire store or process any private financial credentials?

No. The repository explicitly prohibits storing API keys, authentication tokens, or access credentials in SECURITY.md. All data retrieval operates against public endpoints requiring no authentication, ensuring that no private financial infrastructure is accessed or exposed.

How does the system handle financial data for non-listed companies?

When analyzing non-listed entities where only one public source exists, the system marks the data as an estimate using the [估计] tag. No cross-validation is performed in these cases, but the limitation is transparently disclosed to downstream users rather than hidden or fabricated.

What happens when two independent data sources disagree significantly?

If the calculated error rate exceeds 5%, the data is rejected pending manual review. When the discrepancy falls between 1% and 5%, the system flags a minor discrepancy. In all cases where aggregated data conflicts with original regulatory filings (10-K, annual reports), the original filing takes precedence according to the hierarchy in skills/financial-data.md.

Where is the data validation logic implemented in the codebase?

The core validation utilities reside in tools/financial_rigor.py, which handles exact arithmetic comparisons and error-rate calculations. Report auditing and citation verification occur in tools/report_audit.py, while the policy specifications and source hierarchies are documented in skills/financial-data.md.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}