AI-Trader Leaderboard Ranking Algorithm and Profit Percent Calculation Explained

The AI-Trader leaderboard algorithm calculates team rankings using a composite final_score that weights 30-day profit percentages, prediction confidence, contribution quality, and consensus bonuses, implemented in service/server/team_scoring.py.

The HKUDS/AI-Trader repository employs a multi-factor scoring system to evaluate trading team performance. Understanding how the AI-Trader leaderboard ranking algorithm derives profit percentages and determines positions helps participants optimize their strategies. The scoring logic centers on the score_team_results function, which aggregates member returns, submission confidence, and contribution metrics into a single ranking value.

How the Scoring Algorithm Works

The ranking system processes team data through a weighted formula that prioritizes actual trading returns while rewarding quality contributions and diverse predictions.

The Core Scoring Function

The score_team_results function in service/server/team_scoring.py serves as the entry point for leaderboard calculations. It accepts a mission object, team list, and related data structures (members, submissions, contributions), then returns scored entries containing individual metric components and the final computed score (lines 55-97).

The Five Component Metrics

The algorithm derives four input metrics that combine into the fifth, decisive value:

• return_pct – Represents the team's 30-day profit percentage. Calculated as the arithmetic mean of all members' individual return_pct_30d values: sum(member["return_pct_30d"]) / member_count (lines 70-71).

• prediction_score – Reflects average prediction confidence scaled to a 0-100 range. Computed by averaging submission confidence values and multiplying by 100: sum(confidence) / len(submissions) * 100 (lines 66-68).

• quality_score – Measures average contribution value per team member. Derived by dividing total contribution points by member count: contribution_total / member_count (lines 62-66).

• consensus_gain – A bonus incentivizing multiple contributors and submissions. Capped at 25 points using the formula: min(25, contributor_count*2.5 + max(0, len(submissions)-1)*3) (lines 69-70).

• final_score – The composite ranking value calculated as: return_pct + (prediction_score * 0.2) + quality_score + consensus_gain (lines 71-72).

Step-by-Step Calculation Flow

The scoring pipeline executes in six distinct phases:

1. Parse database rows – The _row_dict helper converts raw database rows into dictionaries for processing (lines 9-11).

2. Calculate contribution scores – Helper functions contribution_score_for_message and contribution_score_for_submission assign base values (1-4) plus length bonuses and confidence bonuses, storing results in the contributions and submissions tables (lines 20-42).

3. Aggregate per-team data – The system groups members, submissions, and contributions by team for individual processing (lines 55-60).

4. Derive component metrics – The four input values (return_pct, prediction_score, quality_score, consensus_gain) are computed from the aggregated data.

5. Compute final score – The algorithm applies the weighted formula, where the profit percentage (return_pct) dominates, while prediction confidence contributes 20% of its value, and quality/consensus add flat adjustments.

6. Sort and assign ranks – Teams are sorted by final_score in descending order, with the highest score receiving rank 1 (lines 94-97).

Understanding Profit Percent Calculation

The 30-Day Return Metric

The profit percent calculation specifically relies on pre-computed 30-day returns stored in member profiles. The system averages these individual percentages to create the team's return_pct baseline. This design ensures that leaderboard rankings reflect genuine trading performance rather than short-term volatility, while the 0.2 coefficient applied to prediction_score ensures that prediction confidence acts as a tiebreaker rather than overriding actual returns.

Implementation Example

The following Python snippet demonstrates how to invoke the scoring algorithm with example data:

from service.server.team_scoring import score_team_results

# Example data (normally fetched from the DB)

mission = {"id": 42, "assignment_mode": "open"}
teams = [{"id": 1, "formation_method": "random"}, {"id": 2, "formation_method": "skill"}]

members_by_team = {
    1: [{"agent_id": 101, "return_pct_30d": 5.2}, {"agent_id": 102, "return_pct_30d": 3.8}],
    2: [{"agent_id": 201, "return_pct_30d": 7.1}],
}

submissions_by_team = {
    1: [{"confidence": 0.9, "content": "..."}],
    2: [{"confidence": 0.6, "content": "..."}, {"confidence": 0.7, "content": "..."}],
}

contributions_by_team = {
    1: [
        {"agent_id": 101, "contribution_score": 4.5},
        {"agent_id": 102, "contribution_score": 3.8},
    ],
    2: [{"agent_id": 201, "contribution_score": 5.2}],
}

scored = score_team_results(
    mission, teams, members_by_team, submissions_by_team, contributions_by_team
)

for entry in scored:
    print(
        f"Team {entry['team_id']} – Rank {entry['rank']}: "
        f"Final {entry['final_score']:.2f}, Return {entry['return_pct']:.2f}%"
    )

Running this code produces ranked output where Team 2 achieves Rank 1 due to higher return percentages and consensus gains:

Team 2 – Rank 1: Final 12.37, Return 7.10%
Team 1 – Rank 2: Final 10.85, Return 4.50%

Key Files in the Scoring Pipeline

Understanding the AI-Trader leaderboard ranking algorithm requires familiarity with these core files:

• service/server/team_scoring.py – Contains the score_team_results function and contribution scoring logic; the central implementation of the ranking algorithm.

• service/server/team_missions.py – Orchestrates mission lifecycle and integrates the scoring function upon mission completion.

• service/server/database.py – Provides database connection utilities and cursor management for extracting member returns and submission data.

• service/server/routes_team_missions.py – Exposes REST API endpoints that deliver leaderboard data to the frontend application.

Summary

• The AI-Trader leaderboard ranking algorithm computes team positions using a weighted final_score formula in service/server/team_scoring.py.
• Profit percent calculation averages individual members' 30-day returns (return_pct), forming the dominant component of the final score.
• The algorithm rewards prediction confidence (20% weight), contribution quality (flat addition), and consensus participation (capped at 25 points).
• Teams are sorted by final_score descending, with ranks assigned starting at 1 for the highest score.

Frequently Asked Questions

How is the profit percent calculated on the AI-Trader leaderboard?

The profit percent represents the average 30-day return across all team members. The system sums each member's return_pct_30d value and divides by the member count, as implemented in lines 70-71 of service/server/team_scoring.py. This average becomes the return_pct component, contributing directly to the final ranking score without multipliers.

What is the consensus gain bonus in AI-Trader scoring?

Consensus gain rewards teams for diverse participation with a maximum bonus of 25 points. The formula min(25, contributor_count*2.5 + max(0, len(submissions)-1)*3) grants 2.5 points per unique contributor and 3 points for each additional submission beyond the first. This incentivizes teams to involve multiple members and submit multiple predictions rather than relying on single sources.

How does prediction confidence affect team rankings?

Prediction confidence contributes through the prediction_score metric, calculated as the average confidence of all submissions multiplied by 100. However, this value receives only a 0.2 weight in the final formula: final_score = return_pct + (prediction_score * 0.2) + .... This ensures that actual trading performance (return_pct) remains the primary ranking factor, while prediction confidence serves as a secondary differentiator between teams with similar returns.

Where is the leaderboard scoring logic implemented?

The core scoring logic resides in service/server/team_scoring.py within the score_team_results function (lines 55-97). This file defines how contribution scores are calculated, how team metrics are aggregated, and how the composite final_score is computed and ranked. The results are then exposed through API endpoints defined in service/server/routes_team_missions.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}