How to Add a New Website to Maigret's Database: A Complete Guide

To add a new website to Maigret, create a JSON entry in maigret/resources/data.json defining the site's URL patterns and detection logic, then run python utils/update_site_data.py to regenerate the documentation and metadata.

Maigret is an open-source OSINT tool that automates username checks across hundreds of social networks and websites. The tool stores all supported sites in a centralized JSON database that maps username patterns to HTTP validation logic. Understanding how to extend this database allows you to customize Maigret for specific investigative targets or contribute new sites to the upstream project.

Understanding Maigret's Site Database Architecture

Maigret's site definitions live in maigret/resources/data.json, a structured JSON file bundled with the package. Each top-level key represents a unique site identifier (e.g., "Twitter", "GitHub"), with nested key-value pairs describing how the engine queries and validates accounts.

The core engine reads this JSON at runtime through maigret/sites.py. This module defines a Site class that deserializes each entry into an object with typed attributes mapping directly to JSON keys like url, urlMain, checkType, and regexCheck. When you add a new site, you are essentially extending the schema that this class expects, ensuring the engine can instantiate new Site objects for your target platform.

After modifying data.json, you must run utils/update_site_data.py. This helper script regenerates the human-readable sites.md file and updates maigret/resources/db_meta.json with derived statistics. These generated files power the --list-sites CLI option and project documentation, making this step mandatory for any database change.

Step-by-Step Guide to Adding a New Site

1. Create the Site Entry in data.json

Copy an existing entry from maigret/resources/data.json that uses similar detection logic, then modify it for your target site. At minimum, you must define how to construct the profile URL and how to verify if a username exists.

"ExampleForum": {
    "tags": ["forum", "tech"],
    "urlMain": "https://example.com/",
    "url": "https://example.com/user/{username}",
    "checkType": "status_code",
    "usernameClaimed": "alice",
    "usernameUnclaimed": "nonexistent_user_99999",
    "regexCheck": "^[a-zA-Z0-9_]{3,20}$",
    "alexaRank": 5000
}

Critical fields:

• urlMain: The base domain for display purposes.
• url: The profile URL template using {username} as a placeholder.
• checkType: The detection method—status_code, message, or response_url are common options.
• usernameClaimed / usernameUnclaimed: Known existing and non-existing usernames for self-testing.

2. Regenerate the Documentation

Navigate to the repository root and execute the updater script to synchronize sites.md and metadata:

python utils/update_site_data.py

This command parses your changes to data.json and rebuilds the markdown list of supported sites. If you skip this step, the CLI's --list-sites output will drift out of sync with the actual database.

3. Test Your New Site

Validate that your detection logic works correctly before submitting:

# Test a known existing account

maigret alice -t ExampleForum

# Test a known non-existing account  

maigret nonexistent_user_99999 -t ExampleForum

The first command should report the account as Claimed, while the second should report Unclaimed. If using checkType: message, verify that presenseStrs or absenceStrs correctly trigger based on the HTTP response body.

4. Submit Your Changes

Commit the following files to your fork:

• Modified maigret/resources/data.json
• Regenerated sites.md
• Updated maigret/resources/db_meta.json (auto-generated by the script)
• Optional: Add test cases in the test suite covering the new site's detection logic

Submit a pull request to the soxoj/maigret repository with a clear description of the site and your validation methodology.

Essential Configuration Fields

When you add a new website to Maigret, these fields control the scanning behavior:

• checkType: Determines how Maigret interprets the HTTP response.

  • status_code: Username exists if the request returns a specific status (usually 200).
  • message: Checks for specific strings in the response body using presenseStrs or absenceStrs.
  • response_url: Validates based on URL redirects or final destination.

• regexCheck: A regular expression that validates username format before sending requests (e.g., ^[a-zA-Z0-9]+$ for alphanumeric-only sites).

• headers: Custom HTTP headers required by the target site, such as specific User-Agent strings or Accept-Language values.

• alexaRank: Integer representing the site's global traffic rank, used for prioritizing scans.

• tags: Array of category strings (e.g., ["social"], ["gaming"]) enabling filtered scans via maigret --tags gaming.

Summary

• Maigret stores site definitions in maigret/resources/data.json, which the Site class in maigret/sites.py parses at runtime.
• To add a new website, edit data.json with the appropriate url, checkType, and validation strings, then run python utils/update_site_data.py to regenerate documentation.
• Always test with known claimed and unclaimed usernames using the -t flag before submitting changes.
• The process requires no API keys and supports any site exposing public user profiles via predictable URL patterns.

Frequently Asked Questions

What file format does Maigret use for its site database?

Maigret uses a JSON file located at maigret/resources/data.json. Each site is a top-level object key containing nested configuration fields that the engine reads via the Site class in maigret/sites.py.

Do I need to manually edit the sites.md file when adding a new site?

No. Running python utils/update_site_data.py automatically regenerates sites.md and maigret/resources/db_meta.json based on the current state of data.json. Manual edits to these files will be overwritten.

What is the difference between usernameClaimed and usernameUnclaimed?

These fields provide sample usernames for automated testing. usernameClaimed should be a username known to exist on the platform, while usernameUnclaimed should be impossible or extremely unlikely to exist. Maigret uses these to self-verify that its detection logic remains accurate for the site.

Can I add sites that require custom HTTP headers?

Yes. Include a headers object in your JSON entry with any required header key-value pairs, such as custom User-Agent strings or authentication tokens, and Maigret will include them in all requests to that site.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}