How CodeWiki Handles Different Repository Types: GitHub vs Local Repositories

CodeWiki detects repository types using regex patterns in utils/repository_parser.py, routing GitHub URLs to the GitHub REST API and local paths to an internal /local_repo/structure endpoint, then normalizing both into a unified structure for wiki generation.

CodeWiki, an open-source documentation generator by quangdungluong/codewiki, supports both remote GitHub repositories and local file system directories. Understanding how the codebase distinguishes between these repository types reveals the architecture behind its flexible input handling.

Repository Type Detection in CodeWiki

The system identifies whether a user input points to a web-based GitHub repository or a local folder through pattern matching in utils/repository_parser.py.

Detecting GitHub Web Repositories

CodeWiki recognizes GitHub URLs using the custom_git_regex pattern:

custom_git_regex = r'^(?:https?:\/\/)?([^\/]+)\/(.+?)\/([^\/]+)(?:\.git)?\/?$'

When parse_repository_input matches this pattern, it sets the repository type to "web" and extracts the owner, repo, and full_path components. This metadata enables direct GitHub API calls without additional parsing.

Detecting Local File System Repositories

For local directories, the parser checks for absolute path patterns:

• Windows paths: Matched against ^[a-zA-Z]:\\
• Unix paths: Identified by leading forward slash /

When detected, the parser returns type: "local", stores the absolute path in local_path, and assigns "local" as the owner placeholder. This allows the system to route the request to the internal local repository handler rather than external APIs.

Fetching Repository Structures

The RepositoryStructureFetcher class in utils/repository_structure.py consumes the parsed metadata and implements divergent fetching strategies based on the repository type.

GitHub API Integration for Web Repositories

For web repositories, the fetcher constructs GitHub REST API endpoints:

elif self.repo_info["type"] == "web":
    for branch in ["main", "master"]:
        api_url = f"https://api.github.com/repos/{self.owner}/{self.repo}/git/trees/{branch}?recursive=1"
        response = requests.get(api_url, headers=self.create_github_headers(self.token))
        # Process tree data...

    
    # Fetch README separately

    readme_response = requests.get(
        f"https://api.github.com/repos/{self.owner}/{self.repo}/readme",
        headers=self.create_github_headers(self.token),
    )

The implementation attempts both main and master branches to maximize compatibility, then retrieves the README through the dedicated /readme endpoint. Both operations use the same authentication headers and error handling patterns.

Local Repository Structure Endpoint

For local repositories, the system delegates file system traversal to an internal API endpoint:

if self.repo_info["type"] == "local" and self.repo_info.get("local_path"):
    response = requests.get(
        f"{TARGET_SERVER_BASE_URL}/local_repo/structure?path={urllib.parse.quote(self.repo_info['local_path'])}"
    )
    data = response.json()
    file_tree_data = data["file_tree"]
    readme_content = data["readme"]

This approach isolates file system operations to the server side, returning a JSON payload with file_tree and readme keys that match the structure returned by the GitHub API path. The fetcher normalizes both sources into identical file_tree_data and readme_content variables for downstream wiki generation.

Code Examples

Parsing a GitHub URL

from utils.repository_parser import parse_repository_input

info = parse_repository_input("https://github.com/quangdungluong/codewiki")

# Returns: {

#   'owner': 'quangdungluong',

#   'repo': 'codewiki',

#   'type': 'web',

#   'full_path': 'quangdungluong/codewiki'

# }

Parsing a Local Folder

info = parse_repository_input("/home/user/projects/my-local-repo")

# Returns: {

#   'owner': 'local',

#   'repo': 'my-local-repo',

#   'type': 'local',

#   'local_path': '/home/user/projects/my-local-repo'

# }

Fetching Repository Structure

from utils.repository_structure import RepositoryStructureFetcher

fetcher = RepositoryStructureFetcher(
    repo_info=info,
    repo_url="https://github.com/quangdungluong/codewiki",
    owner=info["owner"],
    repo=info["repo"],
    token=None,
)

# Execute fetch with status callback

await fetcher.fetch_repository_structure(
    lambda task_id, status, message: None,
    "task-123"
)

# Access normalized structure

print(fetcher.wiki_structure)

Key Implementation Files

• utils/repository_parser.py – Detects repository type using regex patterns for GitHub URLs and local file paths, returning normalized metadata dictionaries.

• utils/repository_structure.py – Implements RepositoryStructureFetcher with divergent logic for GitHub API calls versus local repository endpoint requests.

• api/services/github_service.py – Encapsulates GitHub-specific API operations including tree retrieval and README fetching with authentication handling.

• tests/test_repository_structure.py – Validates correct behavior for both web and local repository types through unit tests.

Summary

• Type Detection: CodeWiki uses regex matching in repository_parser.py to classify inputs as either web (GitHub URLs) or local (absolute file paths).

• GitHub Handling: Web repositories trigger direct GitHub REST API calls to /git/trees/{branch} and /readme endpoints, supporting both main and master branches.

• Local Handling: Local paths route to an internal /local_repo/structure endpoint that returns JSON file trees, isolating file system operations server-side.

• Unified Pipeline: Both paths normalize data into identical file_tree_data and readme_content structures for consistent wiki generation downstream.

Frequently Asked Questions

How does CodeWiki determine if an input is a GitHub repository or a local folder?

CodeWiki applies sequential regex checks in utils/repository_parser.py. It first tests for Windows (^[a-zA-Z]:\\) or Unix (^/) absolute paths to identify local repositories. If neither matches, it checks against custom_git_regex (^(?:https?:\/\/)?([^\/]+)\/(.+?)\/([^\/]+)(?:\.git)?\/?$) to detect GitHub URLs. The first matching pattern determines the type field returned in the metadata dictionary.

What GitHub API endpoints does CodeWiki use to fetch repository data?

For web repositories, CodeWiki calls https://api.github.com/repos/{owner}/{repo}/git/trees/{branch}?recursive=1 to retrieve the recursive file tree, attempting both main and master branches for compatibility. It separately fetches the README via https://api.github.com/repos/{owner}/{repo}/readme. Both endpoints support optional authentication tokens passed through Authorization headers in create_github_headers().

How does CodeWiki handle local repositories without exposing file system details to the client?

Local repository handling uses an internal API abstraction. Instead of reading files directly in the parser, CodeWiki sends the absolute path to {TARGET_SERVER_BASE_URL}/local_repo/structure?path={encoded_path}. This endpoint returns a JSON object containing file_tree and readme strings. The RepositoryStructureFetcher processes this response identically to GitHub API data, ensuring the downstream wiki generation pipeline remains agnostic to the repository source.

Can CodeWiki work with private GitHub repositories?

Yes, CodeWiki supports private repositories through token-based authentication. The RepositoryStructureFetcher accepts an optional token parameter that propagates to create_github_headers(), which constructs the Authorization: token {token} header. This header is included in all GitHub API requests for web repositories, allowing access to private repos granted the token has appropriate repo scope permissions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}