# How HelloGitHub Extracts Project URLs and Metadata from GitHub Event Payloads

> Learn how HelloGitHub extracts project URLs and metadata from GitHub Event Payloads. Discover efficient data extraction and enrichment techniques for WatchEvent data.

- Repository: [削微寒/HelloGitHub](https://github.com/521xueweihan/HelloGitHub)
- Tags: internals
- Published: 2026-02-25

---

**The HelloGitHub bot monitors the GitHub Events API to capture WatchEvent payloads, filters them by recency and ownership constraints, extracts user and repository metadata from nested JSON fields, constructs canonical GitHub URLs, and enriches the data with live star counts via secondary API calls.**

The HelloGitHub project automates the discovery of trending open-source repositories by processing real-time GitHub activity. To extract project URLs and metadata from GitHub event payloads, the bot implements a multi-stage pipeline in [`script/github_bot/github_bot.py`](https://github.com/521xueweihan/HelloGitHub/blob/main/script/github_bot/github_bot.py) that transforms raw JSON events into structured project summaries suitable for newsletter generation.

## Understanding the GitHub Events Data Source

The bot polls the `received_events` endpoint (`https://api.github.com/users/{username}/received_events`) to retrieve a stream of public events performed by users the target account follows. Each event is a JSON object conforming to the GitHub Events API schema, containing nested `actor`, `repo`, and `payload` objects that hold the metadata required for URL construction and project identification.

## The Three-Step Extraction Pipeline

The extraction logic in [`script/github_bot/github_bot.py`](https://github.com/521xueweihan/HelloGitHub/blob/main/script/github_bot/github_bot.py) processes each event through three distinct phases: event validation, metadata extraction, and repository enrichment.

### Step 1: Filter Relevant Watch Events

The `check_condition` function (lines 112-118) validates incoming events to ensure they represent meaningful star activity. The function verifies four criteria:

- **Event type** must be `WatchEvent` (a repository star)
- **Action** must be `"started"` (indicating a new star, not an unstar)
- **Recency** within the configured `DAY` threshold (typically 24 hours)
- **Ownership exclusion** to prevent processing the bot's own repositories (`ACCOUNT['username'] not in data['repo']['name']`)

```python
def check_condition(data):
    # Timezone adjustment (UTC+8)

    create_time = datetime.datetime.strptime(
        data['created_at'], "%Y-%m-%dT%H:%M:%SZ") + datetime.timedelta(hours=8)
    recent = create_time >= datetime.datetime.now() - datetime.timedelta(days=DAY)
    
    if data['type'] == 'WatchEvent' and recent:
        if data['payload']['action'] == 'started' and \
           ACCOUNT['username'] not in data['repo']['name']:
            data['date_time'] = create_time.strftime("%Y-%m-%d %H:%M:%S")
            return True
    return False

```

### Step 2: Extract Project URLs and Basic Metadata

Once validated, the bot extracts nested fields from the event payload to construct a `project_info` dictionary. The bot maps GitHub API fields to canonical URLs:

- `actor.login` → `project_info['user']` and `project_info['user_url']`
- `actor.avatar_url` → `project_info['avatar_url']`
- `repo.name` → `project_info['repo_name']` and `project_info['repo_url']`
- `repo.url` → Stored as the API endpoint for enrichment

```python
project_info = {}
project_info['user'] = fi_data['actor']['login']
project_info['user_url'] = f"https://github.com/{project_info['user']}"
project_info['avatar_url'] = fi_data['actor']['avatar_url']
project_info['repo_name'] = fi_data['repo']['name']
project_info['repo_url'] = f"https://github.com/{project_info['repo_name']}"
project_info['date_time'] = fi_data['date_time']

```

### Step 3: Enrich with Repository Metadata

The bot performs a secondary API call to the repository endpoint (`fi_data['repo']['url']`) to fetch live metadata. This step retrieves the current `stargazers_count` and applies a minimum threshold filter (`STARS = 100`) to ensure only sufficiently popular repositories are included in the newsletter.

```python
repo_api = fi_data['repo']['url']  # e.g., https://api.github.com/repos/owner/repo

repo_json = requests.get(repo_api, timeout=2).json()
project_info['repo_stars'] = int(repo_json.get('stargazers_count', -1))

# Threshold check

if project_info['repo_stars'] >= STARS:
    project_info_list.append(project_info)

```

## Complete Pipeline Implementation

The following consolidated example demonstrates the full extraction workflow from event filtering to final list generation:

```python
import datetime
import requests
from operator import itemgetter

DAY = 1
STARS = 100
ACCOUNT = {'username': 'HelloGitHubBot'}

def check_condition(data):
    """Validate WatchEvent within time window and ownership constraints."""
    create_time = datetime.datetime.strptime(
        data['created_at'], "%Y-%m-%dT%H:%M:%SZ") + datetime.timedelta(hours=8)
    recent = create_time >= datetime.datetime.now() - datetime.timedelta(days=DAY)
    
    if data['type'] == 'WatchEvent' and recent:
        if data['payload']['action'] == 'started' and \
           ACCOUNT['username'] not in data['repo']['name']:
            data['date_time'] = create_time.strftime("%Y-%m-%d %H:%M:%S")
            return True
    return False

def extract_project_info(fi_data):
    """Extract URLs and metadata from GitHub event payload."""
    project_info = {}
    project_info['user'] = fi_data['actor']['login']
    project_info['user_url'] = f"https://github.com/{project_info['user']}"
    project_info['avatar_url'] = fi_data['actor']['avatar_url']
    project_info['repo_name'] = fi_data['repo']['name']
    project_info['repo_url'] = f"https://github.com/{project_info['repo_name']}"
    project_info['date_time'] = fi_data['date_time']
    
    # Enrich with live star count

    repo_api = fi_data['repo']['url']
    repo_json = requests.get(repo_api, timeout=2).json()
    project_info['repo_stars'] = int(repo_json.get('stargazers_count', -1))
    
    return project_info

def get_starred_projects(events):
    """Main pipeline: filter, extract, enrich, and threshold."""
    project_info_list = []
    
    for ev in events:
        if check_condition(ev):
            info = extract_project_info(ev)
            if info['repo_stars'] >= STARS:
                project_info_list.append(info)
    
    return sorted(project_info_list, key=itemgetter('repo_stars'), reverse=True)

```

## Summary

- The HelloGitHub bot monitors the `received_events` endpoint to capture real-time GitHub activity.
- **Event filtering** occurs in `check_condition` within [`script/github_bot/github_bot.py`](https://github.com/521xueweihan/HelloGitHub/blob/main/script/github_bot/github_bot.py), validating `WatchEvent` types, recency, and ownership constraints.
- **URL extraction** maps `actor.login` and `repo.name` fields to canonical GitHub URLs (`https://github.com/{user}` and `https://github.com/{repo}`).
- **Metadata enrichment** requires a secondary API call to the repository endpoint to retrieve current `stargazers_count` and apply the minimum threshold of 100 stars.
- The final output is a sorted list of project dictionaries containing user profiles, repository links, avatar URLs, and star counts ready for HTML email rendering.

## Frequently Asked Questions

### What GitHub event type does HelloGitHub monitor to detect starred repositories?

HelloGitHub specifically monitors `WatchEvent` payloads, which GitHub emits when a user stars a repository. The bot checks that the payload action equals `"started"` to ensure it captures new star events rather than unstar actions, as implemented in the `check_condition` function in [`script/github_bot/github_bot.py`](https://github.com/521xueweihan/HelloGitHub/blob/main/script/github_bot/github_bot.py).

### Why does the bot perform a secondary API call after processing the event payload?

The initial `received_events` payload contains limited metadata—primarily actor and repository identifiers—but does not include the current star count. The bot calls the repository API endpoint (`fi_data['repo']['url']`) to fetch the live `stargazers_count`, enabling it to filter repositories by popularity (minimum 100 stars) and sort the final results by star count.

### How does HelloGitHub prevent processing its own repositories?

The `check_condition` function includes an ownership validation that checks if `ACCOUNT['username']` appears within the `data['repo']['name']` string. This ensures the bot ignores stars on its own repositories, preventing self-promotion in the generated newsletter content.

### What URL format does the bot generate for user profiles and repositories?

The bot constructs canonical GitHub URLs by prefixing the extracted login and repository names with `https://github.com/`. Specifically, it generates `https://github.com/{actor.login}` for user profiles and `https://github.com/{repo.name}` for repository pages, storing these in `project_info['user_url']` and `project_info['repo_url']` respectively.