How HelloGitHub Extracts Project URLs and Metadata from GitHub Event Payloads

The HelloGitHub bot monitors the GitHub Events API to capture WatchEvent payloads, filters them by recency and ownership constraints, extracts user and repository metadata from nested JSON fields, constructs canonical GitHub URLs, and enriches the data with live star counts via secondary API calls.

The HelloGitHub project automates the discovery of trending open-source repositories by processing real-time GitHub activity. To extract project URLs and metadata from GitHub event payloads, the bot implements a multi-stage pipeline in script/github_bot/github_bot.py that transforms raw JSON events into structured project summaries suitable for newsletter generation.

Understanding the GitHub Events Data Source

The bot polls the received_events endpoint (https://api.github.com/users/{username}/received_events) to retrieve a stream of public events performed by users the target account follows. Each event is a JSON object conforming to the GitHub Events API schema, containing nested actor, repo, and payload objects that hold the metadata required for URL construction and project identification.

The Three-Step Extraction Pipeline

The extraction logic in script/github_bot/github_bot.py processes each event through three distinct phases: event validation, metadata extraction, and repository enrichment.

Step 1: Filter Relevant Watch Events

The check_condition function (lines 112-118) validates incoming events to ensure they represent meaningful star activity. The function verifies four criteria:

• Event type must be WatchEvent (a repository star)
• Action must be "started" (indicating a new star, not an unstar)
• Recency within the configured DAY threshold (typically 24 hours)
• Ownership exclusion to prevent processing the bot's own repositories (ACCOUNT['username'] not in data['repo']['name'])

def check_condition(data):
    # Timezone adjustment (UTC+8)

    create_time = datetime.datetime.strptime(
        data['created_at'], "%Y-%m-%dT%H:%M:%SZ") + datetime.timedelta(hours=8)
    recent = create_time >= datetime.datetime.now() - datetime.timedelta(days=DAY)
    
    if data['type'] == 'WatchEvent' and recent:
        if data['payload']['action'] == 'started' and \
           ACCOUNT['username'] not in data['repo']['name']:
            data['date_time'] = create_time.strftime("%Y-%m-%d %H:%M:%S")
            return True
    return False

Step 2: Extract Project URLs and Basic Metadata

Once validated, the bot extracts nested fields from the event payload to construct a project_info dictionary. The bot maps GitHub API fields to canonical URLs:

• actor.login → project_info['user'] and project_info['user_url']
• actor.avatar_url → project_info['avatar_url']
• repo.name → project_info['repo_name'] and project_info['repo_url']
• repo.url → Stored as the API endpoint for enrichment

project_info = {}
project_info['user'] = fi_data['actor']['login']
project_info['user_url'] = f"https://github.com/{project_info['user']}"
project_info['avatar_url'] = fi_data['actor']['avatar_url']
project_info['repo_name'] = fi_data['repo']['name']
project_info['repo_url'] = f"https://github.com/{project_info['repo_name']}"
project_info['date_time'] = fi_data['date_time']

Step 3: Enrich with Repository Metadata

The bot performs a secondary API call to the repository endpoint (fi_data['repo']['url']) to fetch live metadata. This step retrieves the current stargazers_count and applies a minimum threshold filter (STARS = 100) to ensure only sufficiently popular repositories are included in the newsletter.

repo_api = fi_data['repo']['url']  # e.g., https://api.github.com/repos/owner/repo

repo_json = requests.get(repo_api, timeout=2).json()
project_info['repo_stars'] = int(repo_json.get('stargazers_count', -1))

# Threshold check

if project_info['repo_stars'] >= STARS:
    project_info_list.append(project_info)

Complete Pipeline Implementation

The following consolidated example demonstrates the full extraction workflow from event filtering to final list generation:

import datetime
import requests
from operator import itemgetter

DAY = 1
STARS = 100
ACCOUNT = {'username': 'HelloGitHubBot'}

def check_condition(data):
    """Validate WatchEvent within time window and ownership constraints."""
    create_time = datetime.datetime.strptime(
        data['created_at'], "%Y-%m-%dT%H:%M:%SZ") + datetime.timedelta(hours=8)
    recent = create_time >= datetime.datetime.now() - datetime.timedelta(days=DAY)
    
    if data['type'] == 'WatchEvent' and recent:
        if data['payload']['action'] == 'started' and \
           ACCOUNT['username'] not in data['repo']['name']:
            data['date_time'] = create_time.strftime("%Y-%m-%d %H:%M:%S")
            return True
    return False

def extract_project_info(fi_data):
    """Extract URLs and metadata from GitHub event payload."""
    project_info = {}
    project_info['user'] = fi_data['actor']['login']
    project_info['user_url'] = f"https://github.com/{project_info['user']}"
    project_info['avatar_url'] = fi_data['actor']['avatar_url']
    project_info['repo_name'] = fi_data['repo']['name']
    project_info['repo_url'] = f"https://github.com/{project_info['repo_name']}"
    project_info['date_time'] = fi_data['date_time']
    
    # Enrich with live star count

    repo_api = fi_data['repo']['url']
    repo_json = requests.get(repo_api, timeout=2).json()
    project_info['repo_stars'] = int(repo_json.get('stargazers_count', -1))
    
    return project_info

def get_starred_projects(events):
    """Main pipeline: filter, extract, enrich, and threshold."""
    project_info_list = []
    
    for ev in events:
        if check_condition(ev):
            info = extract_project_info(ev)
            if info['repo_stars'] >= STARS:
                project_info_list.append(info)
    
    return sorted(project_info_list, key=itemgetter('repo_stars'), reverse=True)

Summary

• The HelloGitHub bot monitors the received_events endpoint to capture real-time GitHub activity.
• Event filtering occurs in check_condition within script/github_bot/github_bot.py, validating WatchEvent types, recency, and ownership constraints.
• URL extraction maps actor.login and repo.name fields to canonical GitHub URLs (https://github.com/{user} and https://github.com/{repo}).
• Metadata enrichment requires a secondary API call to the repository endpoint to retrieve current stargazers_count and apply the minimum threshold of 100 stars.
• The final output is a sorted list of project dictionaries containing user profiles, repository links, avatar URLs, and star counts ready for HTML email rendering.

Frequently Asked Questions

What GitHub event type does HelloGitHub monitor to detect starred repositories?

HelloGitHub specifically monitors WatchEvent payloads, which GitHub emits when a user stars a repository. The bot checks that the payload action equals "started" to ensure it captures new star events rather than unstar actions, as implemented in the check_condition function in script/github_bot/github_bot.py.

Why does the bot perform a secondary API call after processing the event payload?

The initial received_events payload contains limited metadata—primarily actor and repository identifiers—but does not include the current star count. The bot calls the repository API endpoint (fi_data['repo']['url']) to fetch the live stargazers_count, enabling it to filter repositories by popularity (minimum 100 stars) and sort the final results by star count.

How does HelloGitHub prevent processing its own repositories?

The check_condition function includes an ownership validation that checks if ACCOUNT['username'] appears within the data['repo']['name'] string. This ensures the bot ignores stars on its own repositories, preventing self-promotion in the generated newsletter content.

What URL format does the bot generate for user profiles and repositories?

The bot constructs canonical GitHub URLs by prefixing the extracted login and repository names with https://github.com/. Specifically, it generates https://github.com/{actor.login} for user profiles and https://github.com/{repo.name} for repository pages, storing these in project_info['user_url'] and project_info['repo_url'] respectively.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}