How to Run Tests for AI Berkshire: Validating the Research Codebase

AI Berkshire does not ship a traditional unit-test suite; instead, you run tests by executing the audit scripts and tool validation commands that verify research outputs and data integrity.

The xbtlin/ai-berkshire repository organizes its validation workflow around research integrity rather than conventional software testing. Because this is a research-oriented codebase, you won't find a pytest or unittest directory. Instead, learning how to run tests for AI Berkshire means leveraging the built-in report-audit scripts and Codex skill synchronization tools that function as the project's test harness.

Understanding the Testing Architecture

Unlike standard Python libraries, AI Berkshire treats validation as a research audit process. The codebase lacks a dedicated tests/ folder or pytest configuration. Instead, validation occurs through executable scripts that check report formatting, citation integrity, and data consistency. According to the repository structure, the primary validation logic resides in tools/report_audit.py, which serves as the main test harness for verifying generated research outputs.

Step-by-Step Guide to Running Tests

Follow these four steps to validate the AI Berkshire codebase and ensure all components function correctly.

Install the Development Environment

Before running any validation commands, install the Python dependencies required by the project. The repository typically lists these in requirements.txt or pyproject.toml.

pip install -r requirements.txt

Ensure the repository's helper scripts remain executable and that your Python environment includes all financial analysis dependencies referenced by the tool modules.

Execute the Report Audit Tool

The tools/report_audit.py script functions as the primary test suite for the project. This script walks through the reports/ directory and validates formatting, citations, and data consistency issues.

python -m tools.report_audit

When you run this command, it scans generated reports for structural errors and integrity problems, effectively serving as the comprehensive test harness for research outputs.

Synchronize and Validate Codex Skills

After modifying any skill markdown files in the skills/ directory, you must validate the changes by regenerating the Codex skill files. The scripts/sync-codex-skills.py script performs this synchronization and checks for syntax errors, exiting with a non-zero status if validation fails.

python3 scripts/sync-codex-skills.py

This step acts as a critical test for the skill generation pipeline, ensuring that markdown source files in skills/*.md translate correctly into functional Codex skills without syntax errors.

Run Individual Tool Scripts for Sanity Checks

Many analytical utilities contain built-in self-checking logic that raises exceptions on malformed data. Running these scripts with sample inputs serves as ad-hoc testing. For example, you can validate the financial rigor calculations by importing calculate_rigour from tools/financial_rigor.py.

from tools.financial_rigor import calculate_rigour
print(calculate_rigour([100, 105, 110]))

Similarly, the momentum backtest tools (tools/momentum_backtest.py and tools/momentum_backtest_v2.py) perform internal validation and raise errors if input data lacks required integrity.

Key Files for Test Validation

Understanding these specific file paths helps you navigate the validation process:

• tools/report_audit.py - Walks through reports/ and verifies format, citations, and data consistency; this is the primary test-like validator.
• scripts/sync-codex-skills.py - Regenerates Codex skill files from skills/*.md and checks for syntax errors; failures act as test failures.
• tools/financial_rigor.py - Contains reusable financial calculations including the calculate_rigour function with built-in sanity checks.
• tools/momentum_backtest.py and tools/momentum_backtest_v2.py - Perform momentum-based backtests and raise errors if input data is malformed.
• skills/ - Directory containing source markdown files for Claude Code and Codex skills; changes here must be validated by the sync script.

Adding Formal Unit Tests

If you require a traditional testing framework, you can extend the repository by creating standard pytest tests that import the tool modules and assert on their return values. Create a tests/ directory and write Python files that import functions from tools/financial_rigor or other modules until such a suite becomes part of the main repository.

Summary

• AI Berkshire uses audit scripts rather than traditional unit tests for validation.
• Run python -m tools.report_audit to verify report integrity and formatting.
• Execute python3 scripts/sync-codex-skills.py after editing files in skills/ to validate Codex skill generation.
• Use individual tool scripts like tools/financial_rigor.py for ad-hoc validation of specific calculations.
• The repository validates functionality through research output integrity rather than software assertion libraries.

Frequently Asked Questions

Is there a pytest test suite for AI Berkshire?

No. The xbtlin/ai-berkshire repository does not ship with pytest or unittest files. Instead, the project relies on the tools/report_audit.py script and skill synchronization commands to validate functionality and data integrity.

What does the report_audit.py script check?

The tools/report_audit.py script walks through the reports/ directory and verifies formatting standards, citation integrity, and data consistency. It functions as the primary test harness by scanning generated research outputs for structural errors and validation failures.

How do I validate changes to skill files?

After modifying any markdown files in the skills/ directory, run python3 scripts/sync-codex-skills.py. This script regenerates the Codex skill files and validates that the syntax is correct, exiting with a non-zero status code if it encounters errors in the source markdown.

Can I add formal unit tests to the repository?

Yes. While AI Berkshire currently lacks a formal test suite, you can create one by adding a tests/ directory and writing pytest scripts that import functions from modules like tools/financial_rigor.py and assert on their return values, such as testing the calculate_rigour function with known inputs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}