How to Use the report_audit.py Tool for Pre-Publication Verification

The report_audit.py tool implements a three-step audit workflow—extract, verify, and verdict—to ensure research reports contain only financial figures verified against trusted external sources like Macrotrends and StockAnalysis.

The report_audit.py utility in the xbtlin/ai-berkshire repository is a zero-dependency Python script located at tools/report_audit.py designed to prevent publication of inaccurate financial data. Utilizing high-precision arithmetic from tools/financial_rigor.py, it parses Markdown research reports, randomly samples approximately 15% of detectable financial figures, and validates them against authoritative sources with a strict 1% tolerance threshold before granting a PASS or FAIL verdict.

Step 1: Extract Financial Data Points

The first step generates a randomized sample of financial claims from your report. Run the extraction command:

python3 tools/report_audit.py extract \
  --report reports/your-report-name.md \
  --ratio 0.15 \
  --seed 42 \
  --dry-run

The extract command invokes extract_data_points() (lines 55-66) to locate numerical values in tables, "label: value" rows, and bold-number lines. It then calls sample_points() (lines 29-36) to randomly select a subset of data points, defaulting to 15% of the total with a minimum of 3 and maximum of 30 points to ensure statistical coverage without excessive manual labor.

Use --dry-run to preview the selected points without generating output. Omitting this flag produces a JSON template to stdout containing entries with placeholders for fetched_value and fetched_source. Each entry includes metadata such as id, label, reported_value, unit, line_number, and raw_text to facilitate precise verification.

Step 2: Populate the Verification Template

After extraction, manually verify each sampled data point against trusted sources. The JSON template structure appears as follows:

[
  {
    "id": 1,
    "label": "营业收入 · 2023年",
    "reported_value": 7518,
    "unit": "亿",
    "line_number": 123,
    "raw_text": "营业收入：7518亿元",
    "fetched_value": null,
    "fetched_source": "",
    "fetched_value2": null,
    "fetched_source2": ""
  }
]

Fill the following fields for each entry:

• fetched_value: The primary verification value from sources like Macrotrends or StockAnalysis
• fetched_source: Name of the primary data source
• fetched_value2 (optional): Secondary verification value for cross-reference
• fetched_source2 (optional): Name of the secondary source

Save the completed JSON to a file (e.g., verification_results.json) for the final step.

Step 3: Render the Final Verdict

Submit the populated verification data to receive a PASS or FAIL judgment:

python3 tools/report_audit.py verdict \
  --results "$(cat verification_results.json)" \
  --report "My Research Report" \
  --output-json

The verdict command executes render_verdict() (lines 53-70), which calculates relative percentage differences using _pct_diff() (lines 46-51). It applies a 1% tolerance (_TOLERANCE = 0.01 at line 43) to classify each data point as:

• PASS: Fetched value matches reported value within 1%
• FAIL: Deviation exceeds 1%
• WARN: Primary source passes but secondary source diverges significantly

The command outputs a color-coded table summarizing each item's status. With --output-json, it emits a machine-readable verdict dictionary (line 111) containing verdict, pass_count, fail_count, and warn_count. The script exits with status 0 for PASS and 1 for FAIL (line 115), enabling integration with CI pipelines.

Automating the Verification Workflow

Combine all three steps into a shell script for automated pre-publication checks:

#!/usr/bin/env bash
set -euo pipefail

REPORT="reports/tesla/tesla-research-2023.md"

# Extract data points

JSON_TEMPLATE=$(python3 tools/report_audit.py extract \
  --report "$REPORT" \
  --ratio 0.15 \
  --seed 123)

# Populate verification data (manual step simulated here)

FILLED=$(echo "$JSON_TEMPLATE" | jq 'map(
  .fetched_value = 1234,
  .fetched_source = "macrotrends",
  .fetched_value2 = 1230,
  .fetched_source2 = "stockanalysis"
)')

# Render verdict

python3 tools/report_audit.py verdict \
  --results "$FILLED" \
  --report "Tesla Research – 2023"

This automation ensures every numerical claim is cross-checked against external sources before publication, with the non-zero exit code blocking merges if verification fails.

Summary

• Three-step workflow: Extract financial data with extract, manually verify against trusted sources, then render a verdict
• Statistical sampling: By default samples 15% of data points (minimum 3, maximum 30) using sample_points() to balance thoroughness with efficiency
• Strict tolerance: The _TOLERANCE = 0.01 constant enforces a 1% maximum deviation between reported and fetched values
• CI-ready: Exit codes (0 for PASS, 1 for FAIL) and --output-json support seamless integration into publication pipelines
• Zero dependencies: The tool requires only Python 3 standard library, leveraging tools/financial_rigor.py for precision arithmetic

Frequently Asked Questions

What is the 1% tolerance rule in report_audit.py?

The tool defines _TOLERANCE = 0.01 at line 43 of tools/report_audit.py, meaning fetched values must fall within 1% of the reported figure to receive a PASS classification. This accounts for minor rounding differences or timing discrepancies between data sources while flagging material errors.

How many data points does the tool sample?

The sample_points() function (lines 29-36) selects approximately 15% of all detected financial figures by default, with a hard minimum of 3 points and maximum of 30. You can adjust the proportion using the --ratio parameter, though the min/max constraints remain to ensure statistical validity without excessive manual verification.

Can report_audit.py integrate with CI/CD pipelines?

Yes. The script exits with status code 0 when all sampled points PASS and 1 when any point FAILs (line 115). Combined with the --output-json flag for machine-readable results, you can embed the three-step workflow in GitHub Actions, GitLab CI, or other automation platforms to block publication of unverified reports.

What file formats does the audit tool support?

The extract_data_points() function (lines 55-66) specifically parses Markdown report files (.md) to identify numerical values in tables, labeled rows, and bold text. The tool outputs JSON templates and accepts JSON verification data, making it compatible with standard Unix pipelines and automation tools.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}