How to Debug AI-Berkshire Agents and Skills: A Step-by-Step Guide

Debugging AI-Berkshire agents requires validating skill markdown definitions, inspecting parallel Agent execution logs, and manually testing downstream tools like tools/financial_rigor.py.

AI-Berkshire is an open-source investment research framework built around three layers—Skill, Agent, and Tool—that work together to produce reproducible financial analysis. When you invoke a skill (for example, /investment-team), the system parses the skill’s markdown definition, creates a Team object, and launches parallel Agent processes via the internal Task tool. Understanding how to debug AI-Berkshire agents and skills ensures that each component—from prompt templates to financial calculations—executes correctly.

Understanding the AI-Berkshire Architecture

Before diving into specific debugging steps, it helps to understand how the three layers interact according to the xbtlin/ai-berkshire source code.

• Skill Layer: Defined in skills/*.md files (e.g., skills/investment-team.md), these markdown files declare required arguments, execution stages, and Agent prompt templates.
• Agent Layer: Each Agent runs as a parallel process managed by the Task tool. Agents perform web searches, fetch data, and stream progress messages back to the Team Lead.
• Tool Layer: Post-processing scripts like tools/financial_rigor.py validate calculations such as market-cap verification and valuation checks.

When a skill is invoked, the Team Lead coordinates Agents, aggregates outputs, and applies tool-layer validation before assembling the final report.

Step 1: Validate the Skill Definition

Start debugging by examining the skill's canonical definition. Each skill in AI-Berkshire is declared as a markdown file under the skills/ directory.

Open the relevant .md file (for example, skills/investment-team.md) and verify:

• Argument patterns: Ensure the required arguments match the command you intend to run.
• Stage sequences: Confirm that "Stage X" sections correctly reference the intended Agent prompts.
• Placeholder syntax: Look for variables like $ARGUMENTS that get substituted at runtime.

If you modify a skill file, regenerate the Codex artifacts to ensure consistency:

python3 scripts/sync-codex-skills.py --check

This script validates that the generated codex-skills/*/SKILL.md files remain synchronized with your source changes.

Step 2: Inspect the Agent Workflow

Each parallel Agent is created and managed by the internal Task tool. To debug Agent execution, enable verbose logging to see exactly what prompts are sent and how Agents respond.

Enable Debug Mode

Run your skill with the --debug flag to print the full prompt sent to each Agent:

codex --debug /investment-team 腾讯

If using Claude Code, use:

claude --dangerously-skip-permissions /investment-team 腾讯

Analyze Task Logs

When --debug is active, the console shows:

• Agent IDs and start times
• Web search results from WebSearch calls
• Status messages (completed, failed)
• Error messages such as "WebSearch limit exceeded" or "Prompt size > limit"

The skill markdown contains a "启动 N 个并行 Agent" section that defines the prompt template for each Agent. Cross-reference the debug output against this template to verify that placeholders like $ARGUMENTS and 信息丰富度评级 are correctly substituted.

Isolate Failing Agents

If a specific Agent fails or produces poor output:

1. Copy the full prompt from the debug log.
2. Paste it into a local Claude or ChatGPT session.
3. Verify that the LLM returns a valid answer with the provided context.

This technique isolates whether the issue stems from the prompt template itself or the execution environment.

Step 3: Check the Tool Layer

After Agents complete their work, the Team Lead may invoke helper scripts for final validation. These tools reside in the tools/ directory and perform precise calculations that LLMs might handle inconsistently.

Manually Execute Financial Validation

Run helper scripts directly to verify calculations. For example, to check market-cap calculations:

python3 tools/financial_rigor.py verify-market-cap \
  --price 510 --shares 9.11e9 --reported 4.65e12 --currency HKD

For valuation checks:

python3 tools/financial_rigor.py verify-valuation \
  --price 620 --shares 6.8e9 --pe 14 --currency HKD

The tools/financial_rigor.py script relies on Python's decimal module for precise arithmetic. Running these commands manually confirms that rounding errors or currency conversion issues are not corrupting the final report.

Typical Debugging Workflow

Follow this systematic approach when troubleshooting AI-Berkshire skills:

1. Open the skill markdown (skills/*.md): Verify argument patterns match your CLI command and that stage sections reference correct Agent prompts.
2. Run with --debug or --trace: Confirm generated Agent prompts contain expected placeholders like $ARGUMENTS and 信息丰富度评级.
3. Observe Task logs: Confirm each Agent starts, finishes, and returns a valid status. Watch for errors like "WebSearch limit exceeded" or timeouts.
4. Re-run failing prompts in isolation: Copy debug output prompts into a local LLM session to test response quality.
5. Execute tool scripts manually: Run tools/financial_rigor.py with the same arguments used in the report to verify calculation accuracy.
6. Regenerate Codex artifacts: After editing skills/*.md, run python3 scripts/sync-codex-skills.py --check to update generated files.

Common Pitfalls

Watch for these specific issues when debugging AI-Berkshire agents:

• Missing placeholder substitution: If CLI calls omit required arguments, Agents receive empty strings for $ARGUMENTS, causing generic or incorrect responses.
• Incorrect information-richness rating: Skills assign "A/B/C" ratings to Agents based on data availability; inaccurate ratings change search depth and may return incomplete results.
• Tool version mismatch: tools/financial_rigor.py requires the decimal module. Python environments that fall back to float will cause subtle calculation errors. Verify your interpreter matches the repository's CI configuration.
• Parallel-Task timeout: The default Task timeout is 300 seconds. Heavy web-search workloads may exceed this limit. Increase the timeout via the --timeout flag when testing long-running skills.

Summary

• AI-Berkshire uses a three-layer architecture: Skills (skills/*.md), Agents (via Task tool), and Tools (tools/financial_rigor.py).
• Validate skill definitions by checking markdown files for correct argument patterns and stage sequences.
• Debug Agent workflows using --debug flags to inspect full prompts and Task logs for execution status.
• Verify calculations by manually running tools/financial_rigor.py scripts to isolate financial logic errors.
• Regenerate artifacts with scripts/sync-codex-skills.py after modifying skill definitions.

Frequently Asked Questions

How do I enable debug mode for AI-Berkshire skills?

Use the --debug flag when running skills through Codex (codex --debug /skill-name arguments) or the --dangerously-skip-permissions flag with Claude Code. This prints the full Agent prompts, WebSearch results, and execution status to the console, allowing you to verify that placeholder substitution and Agent coordination work correctly.

What should I check if an Agent returns incomplete results?

First, verify the information-richness rating (A/B/C) assigned in the skill markdown, as this determines search depth. Then check the --debug logs for "WebSearch limit exceeded" messages or prompt size warnings. Finally, copy the exact prompt from the debug log and test it in isolation to determine if the issue is the prompt template or the data sources.

How can I verify financial calculations in the final report?

Manually execute the validation scripts found in tools/financial_rigor.py. For example, run python3 tools/financial_rigor.py verify-market-cap with the same price, shares, and currency values used in the original analysis. This bypasses the LLM and uses Python's decimal module to confirm that market-cap and valuation figures are mathematically correct.

Where are skill definitions stored in the repository?

Canonical skill definitions reside in the skills/ directory (e.g., skills/investment-team.md). These markdown files declare arguments, Agent prompts, and execution stages. The scripts/sync-codex-skills.py script processes these files to generate codex-skills/*/SKILL.md artifacts consumed by the Codex runtime. Refer to AGENTS.md for the high-level architecture documentation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}