Information Richness Rating Mechanism in AI Berkshire: How It Prevents Overconfidence

The AI Berkshire framework assigns A/B/C Information Richness Ratings based on data cross-validation, automatically injecting these grades into research outputs to prevent overconfident conclusions when source material is limited.

The xbtlin/ai-berkshire repository implements an information richness rating mechanism that evaluates the reliability of financial data before generating investment research. This system automatically tags every output with an A, B, or C grade according to source independence and data completeness, serving as a critical guardrail against AI overconfidence and the "more data = more certainty" cognitive illusion.

Understanding the A/B/C Rating System

According to README_EN.md, the framework calculates ratings automatically by analyzing how much reliable, cross-validated data is available for the target company.

A-Rating (Abundant Data)

An A-rating indicates the analysis has abundant, high-quality data from at least two independent sources, with most financial line items tagged with high confidence levels.

B-Rating (Moderate Data)

A B-rating signifies moderate data availability where key metrics are present but derived from limited sources or include estimates explicitly flagged with confidence levels. The documentation specifically notes: "Pop Mart rated B: limited data, estimated metrics flagged with confidence levels."

C-Rating (Sparse Data)

A C-rating represents very sparse data where many figures rely on single sources or rough approximations, resulting in outputs containing "gray-zone" or "unknown" tags.

How the Rating Prevents Overconfidence

The rating mechanism prevents overconfidence through three specific technical implementations in the research pipeline.

Confidence-Aware Presentation

When skills like /investment-research or /investment-memo-craft generate reports, they inject the richness rating alongside standard sections (business quality, moat, valuation). As implemented in codex-skills/investment-memo-craft/SKILL.md at line 98, numbers display with confidence icons (🟢 high, 🟡 medium, 🔴 low), reminding readers that overall certainty may be limited despite narrative length.

Bias Mitigation

The rating directly confronts the cognitive bias where longer narratives create false confidence. By surfacing the data foundation rather than the polish of the output, the framework forces users to evaluate source quality. This workflow is detailed in skills/investment-research.md.

Decision Gating

Downstream skills such as /investment-checklist and /investment-team automatically downgrade "Pass" recommendations to "Conditional" or "Gray Zone" when the richness rating falls below A. This decision gating, supported by calculations in tools/financial_rigor.py, stops the system from issuing over-confident verdicts on poorly sourced analyses.

Implementation in the Codebase

The mechanism spans multiple components. The exact-precision calculations feeding confidence assessments reside in tools/financial_rigor.py, while the research workflow that computes and displays ratings is implemented in skills/investment-research.md.

When executing the research command:

> /investment-research PopMart

The system outputs:

-------------------------------------------------
Information Richness Rating: **B**
Data Sources: 2 independent financial statements, 1 analyst report
Confidence Flags:
  • Revenue (2024) – 🟡 Medium confidence – estimate from limited filings
  • Market Cap – 🟢 High confidence – cross‑validated with Bloomberg & HKEX
-------------------------------------------------
Conclusion: Pass with caution – see “B” rating note above.

For programmatic access, the structured response follows this pattern:

{
  "richness_rating": "B",
  "confidence": {
    "revenue": "medium",
    "market_cap": "high"
  },
  "recommendation": "conditional"
}

Summary

• The information richness rating mechanism assigns A/B/C grades based on data quality and source independence
• A-rated analyses require ≥2 independent sources with high-confidence line items
• B and C ratings trigger explicit confidence flags and "gray-zone" tags
• Downstream skills automatically downgrade recommendations when ratings fall below A
• The system uses visual confidence icons (🟢🟡🔴) and structured JSON output to communicate uncertainty

Frequently Asked Questions

What distinguishes an A rating from a B or C rating?

An A rating requires abundant, high-quality data from at least two independent sources with most financial line items showing high confidence, while a B rating indicates moderate data with limited sources or flagged estimates, and a C rating indicates very sparse data with single-source figures and extensive "unknown" tags.

How does the mechanism prevent overconfidence in practice?

The framework surfaces the rating alongside narrative outputs to force evaluation of the data foundation rather than content polish, and downstream skills automatically downgrade "Pass" recommendations to "Conditional" when ratings fall below A, directly countering the illusion that more text equals more certainty.

Which source files implement the rating system?

The rating taxonomy and anti-bias mechanisms are defined in README_EN.md, the memo generation logic resides in codex-skills/investment-memo-craft/SKILL.md (line 98), the research workflow is in skills/investment-research.md, and the exact-precision calculations supporting confidence assessments are in tools/financial_rigor.py.

How do downstream skills react to different rating levels?

Skills such as /investment-checklist and /investment-team use the rating as a decision gate, automatically converting "Pass" recommendations to "Conditional" or "Gray Zone" when the richness rating is B or C, preventing the system from issuing confident verdicts on poorly sourced analyses.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}