# How CodeWiki Handles Different Repository Types: GitHub vs Local Repositories

> Learn how CodeWiki processes GitHub and local repositories. Discover its regex-based type detection and unified structure normalization for seamless wiki generation.

- Repository: [Luong Quang Dung/codewiki](https://github.com/quangdungluong/codewiki)
- Tags: how-to-guide
- Published: 2026-02-16

---

**CodeWiki detects repository types using regex patterns in [`utils/repository_parser.py`](https://github.com/quangdungluong/codewiki/blob/main/utils/repository_parser.py), routing GitHub URLs to the GitHub REST API and local paths to an internal `/local_repo/structure` endpoint, then normalizing both into a unified structure for wiki generation.**

CodeWiki, an open-source documentation generator by `quangdungluong/codewiki`, supports both remote GitHub repositories and local file system directories. Understanding how the codebase distinguishes between these repository types reveals the architecture behind its flexible input handling.

## Repository Type Detection in CodeWiki

The system identifies whether a user input points to a web-based GitHub repository or a local folder through pattern matching in [`utils/repository_parser.py`](https://github.com/quangdungluong/codewiki/blob/main/utils/repository_parser.py).

### Detecting GitHub Web Repositories

CodeWiki recognizes GitHub URLs using the `custom_git_regex` pattern:

```python
custom_git_regex = r'^(?:https?:\/\/)?([^\/]+)\/(.+?)\/([^\/]+)(?:\.git)?\/?$'

```

When `parse_repository_input` matches this pattern, it sets the repository `type` to `"web"` and extracts the `owner`, `repo`, and `full_path` components. This metadata enables direct GitHub API calls without additional parsing.

### Detecting Local File System Repositories

For local directories, the parser checks for absolute path patterns:

- **Windows paths**: Matched against `^[a-zA-Z]:\\`
- **Unix paths**: Identified by leading forward slash `/`

When detected, the parser returns `type: "local"`, stores the absolute path in `local_path`, and assigns `"local"` as the owner placeholder. This allows the system to route the request to the internal local repository handler rather than external APIs.

## Fetching Repository Structures

The `RepositoryStructureFetcher` class in [`utils/repository_structure.py`](https://github.com/quangdungluong/codewiki/blob/main/utils/repository_structure.py) consumes the parsed metadata and implements divergent fetching strategies based on the repository type.

### GitHub API Integration for Web Repositories

For web repositories, the fetcher constructs GitHub REST API endpoints:

```python
elif self.repo_info["type"] == "web":
    for branch in ["main", "master"]:
        api_url = f"https://api.github.com/repos/{self.owner}/{self.repo}/git/trees/{branch}?recursive=1"
        response = requests.get(api_url, headers=self.create_github_headers(self.token))
        # Process tree data...

    
    # Fetch README separately

    readme_response = requests.get(
        f"https://api.github.com/repos/{self.owner}/{self.repo}/readme",
        headers=self.create_github_headers(self.token),
    )

```

The implementation attempts both `main` and `master` branches to maximize compatibility, then retrieves the README through the dedicated `/readme` endpoint. Both operations use the same authentication headers and error handling patterns.

### Local Repository Structure Endpoint

For local repositories, the system delegates file system traversal to an internal API endpoint:

```python
if self.repo_info["type"] == "local" and self.repo_info.get("local_path"):
    response = requests.get(
        f"{TARGET_SERVER_BASE_URL}/local_repo/structure?path={urllib.parse.quote(self.repo_info['local_path'])}"
    )
    data = response.json()
    file_tree_data = data["file_tree"]
    readme_content = data["readme"]

```

This approach isolates file system operations to the server side, returning a JSON payload with `file_tree` and `readme` keys that match the structure returned by the GitHub API path. The fetcher normalizes both sources into identical `file_tree_data` and `readme_content` variables for downstream wiki generation.

## Code Examples

### Parsing a GitHub URL

```python
from utils.repository_parser import parse_repository_input

info = parse_repository_input("https://github.com/quangdungluong/codewiki")

# Returns: {

#   'owner': 'quangdungluong',

#   'repo': 'codewiki',

#   'type': 'web',

#   'full_path': 'quangdungluong/codewiki'

# }

```

### Parsing a Local Folder

```python
info = parse_repository_input("/home/user/projects/my-local-repo")

# Returns: {

#   'owner': 'local',

#   'repo': 'my-local-repo',

#   'type': 'local',

#   'local_path': '/home/user/projects/my-local-repo'

# }

```

### Fetching Repository Structure

```python
from utils.repository_structure import RepositoryStructureFetcher

fetcher = RepositoryStructureFetcher(
    repo_info=info,
    repo_url="https://github.com/quangdungluong/codewiki",
    owner=info["owner"],
    repo=info["repo"],
    token=None,
)

# Execute fetch with status callback

await fetcher.fetch_repository_structure(
    lambda task_id, status, message: None,
    "task-123"
)

# Access normalized structure

print(fetcher.wiki_structure)

```

## Key Implementation Files

- **[`utils/repository_parser.py`](https://github.com/quangdungluong/codewiki/blob/main/utils/repository_parser.py)** – Detects repository type using regex patterns for GitHub URLs and local file paths, returning normalized metadata dictionaries.

- **[`utils/repository_structure.py`](https://github.com/quangdungluong/codewiki/blob/main/utils/repository_structure.py)** – Implements `RepositoryStructureFetcher` with divergent logic for GitHub API calls versus local repository endpoint requests.

- **[`api/services/github_service.py`](https://github.com/quangdungluong/codewiki/blob/main/api/services/github_service.py)** – Encapsulates GitHub-specific API operations including tree retrieval and README fetching with authentication handling.

- **[`tests/test_repository_structure.py`](https://github.com/quangdungluong/codewiki/blob/main/tests/test_repository_structure.py)** – Validates correct behavior for both web and local repository types through unit tests.

## Summary

- **Type Detection**: CodeWiki uses regex matching in [`repository_parser.py`](https://github.com/quangdungluong/codewiki/blob/main/repository_parser.py) to classify inputs as either `web` (GitHub URLs) or `local` (absolute file paths).

- **GitHub Handling**: Web repositories trigger direct GitHub REST API calls to `/git/trees/{branch}` and `/readme` endpoints, supporting both `main` and `master` branches.

- **Local Handling**: Local paths route to an internal `/local_repo/structure` endpoint that returns JSON file trees, isolating file system operations server-side.

- **Unified Pipeline**: Both paths normalize data into identical `file_tree_data` and `readme_content` structures for consistent wiki generation downstream.

## Frequently Asked Questions

### How does CodeWiki determine if an input is a GitHub repository or a local folder?

CodeWiki applies sequential regex checks in [`utils/repository_parser.py`](https://github.com/quangdungluong/codewiki/blob/main/utils/repository_parser.py). It first tests for Windows (`^[a-zA-Z]:\\`) or Unix (`^/`) absolute paths to identify local repositories. If neither matches, it checks against `custom_git_regex` (`^(?:https?:\/\/)?([^\/]+)\/(.+?)\/([^\/]+)(?:\.git)?\/?$`) to detect GitHub URLs. The first matching pattern determines the `type` field returned in the metadata dictionary.

### What GitHub API endpoints does CodeWiki use to fetch repository data?

For web repositories, CodeWiki calls `https://api.github.com/repos/{owner}/{repo}/git/trees/{branch}?recursive=1` to retrieve the recursive file tree, attempting both `main` and `master` branches for compatibility. It separately fetches the README via `https://api.github.com/repos/{owner}/{repo}/readme`. Both endpoints support optional authentication tokens passed through `Authorization` headers in `create_github_headers()`.

### How does CodeWiki handle local repositories without exposing file system details to the client?

Local repository handling uses an internal API abstraction. Instead of reading files directly in the parser, CodeWiki sends the absolute path to `{TARGET_SERVER_BASE_URL}/local_repo/structure?path={encoded_path}`. This endpoint returns a JSON object containing `file_tree` and `readme` strings. The `RepositoryStructureFetcher` processes this response identically to GitHub API data, ensuring the downstream wiki generation pipeline remains agnostic to the repository source.

### Can CodeWiki work with private GitHub repositories?

Yes, CodeWiki supports private repositories through token-based authentication. The `RepositoryStructureFetcher` accepts an optional `token` parameter that propagates to `create_github_headers()`, which constructs the `Authorization: token {token}` header. This header is included in all GitHub API requests for web repositories, allowing access to private repos granted the token has appropriate `repo` scope permissions.