How the loyalsoldier/geoip Repository Generates GeoIP Format Files Weekly: Internal Process and Dependencies

The loyalsoldier/geoip repository automates GeoIP file generation every Thursday at 00:00 UTC through a GitHub Actions workflow that downloads MaxMind GeoLite2 data, compiles a Go 1.25 binary, and executes a plugin-based conversion pipeline to produce multiple output formats including V2Ray DAT and MaxMind MMDB.

The loyalsoldier/geoip project maintains an open-source pipeline for building customizable GeoIP databases used by proxy tools like V2Ray, Clash, and sing-box. Understanding the internal process for generating GeoIP format files weekly reveals how the repository transforms raw MaxMind CSV data into optimized distribution files without manual intervention.

Weekly Automation Workflow

The entire generation process is orchestrated by .github/workflows/build.yml, which triggers automatically via cron: "0 0 * * 4"—midnight UTC every Thursday.

Pipeline Stages

The workflow executes eleven distinct stages to ensure deterministic, reproducible builds:

• Repository Checkout — Uses actions/checkout@v6 to pull the full git history required for release management.

• Go Environment Setup — Installs Go 1.25 (as declared in go.mod) using actions/setup-go@v6.

• Timestamp Generation — Creates a timestamp variable used for release tags and asset versioning.

• MaxMind Data Acquisition — Downloads the latest GeoLite2-Country and GeoLite2-ASN archives using the secret MAXMIND_GEOLITE2_LICENSE via authenticated curl requests.

• File Preparation — Unpacks tar.gz and zip archives, copying .mmdb files and CSV datasets into ./output/ and ./geolite2/ directories using standard Bash utilities.

• Binary Compilation and Conversion — Executes go build ./ to compile the CLI, then runs ./geoip convert -c ./config.json to process all configured inputs and outputs.

• MMDB Integrity Verification — Installs github.com/maxmind/mmdbverify and validates each generated .mmdb file against MaxMind specifications.

• Checksum Generation — Computes SHA-256 hashes for all .dat and .mmdb files, writing them to *.sha256sum files.

• Release Branch Deployment — Commits generated assets to a lightweight release branch used exclusively for GitHub Release attachments.

• CDN Cache Purging — Sends HTTP PURGE requests to jsDelivr for every new asset, ensuring global mirrors refresh immediately.

• Artifact Publication — Uses svenstaro/upload-release-action@v2 to publish final assets as a timestamped GitHub Release, removing intermediate CSV files and raw archives beforehand to keep the distribution clean.

Core Conversion Architecture

The conversion logic centers on a command-line interface defined in main.go and convert.go, implementing a modular plugin system that decouples data ingestion from format output.

CLI Entry Points

In main.go, the application registers the convert sub-command. The convert.go file implements this command, parsing the -c or --config flag to load config.json and orchestrating the transformation pipeline.

Plugin-Based Data Flow

The architecture implements two primary interfaces:

• Input Plugins — Types like maxmindGeoLite2CountryCSV (in plugin/maxmind/maxmind_country_csv_in.go), text (plugin/plaintext/text_in.go), json (plugin/special/json_in.go), and private (plugin/special/private.go) implement the InputConverter interface. Each plugin calls container.Add or container.Remove methods to populate a central data structure defined in lib/container.go and lib/entry.go.

• Output Plugins — Types like v2rayGeoIPDat (plugin/v2ray/dat_out.go), maxmindMMDB (plugin/maxmind/maxmind_country_mmdb_out.go), singboxSRS, and clashRuleSet implement the OutputConverter interface, serializing the container contents into target formats using format-specific writers.

The container maintains CIDR lists and supports add/remove operations, enabling complex transformations like merging MaxMind data with custom IP lists while excluding specific ranges.

Critical Go Dependencies

All third-party modules are declared in go.mod. The generation pipeline relies on specialized libraries for compression, IP handling, and MaxMind compatibility:

• github.com/klauspost/compress — Handles ZIP/ZIP-64 operations for archive extraction and V2Ray DAT output compression.

• github.com/maxmind/mmdbwriter — Writes MaxMind MMDB files via the maxmindMMDB output plugin, including the mmdbtype subpackage for data types.

• github.com/oschwald/geoip2-golang/v2 — Reads original MaxMind .mmdb files when the pipeline requires additional metadata from source databases.

• github.com/spf13/cobra — Provides the CLI framework powering the geoip convert command structure.

• github.com/tailscale/hujson — Parses JSON configuration files while preserving comments, used by lib/config.go for human-readable configuration.

• github.com/tidwall/gjson — Enables fast extraction of IP ranges from remote JSON sources like Google and Fastly endpoint lists.

• go4.org/netipx — Supplies advanced IP address utilities for CIDR merging and prefix manipulation.

• google.golang.org/protobuf — Serializes V2Ray GeoIP protobuf messages in dat_out.go.

• gopkg.in/yaml.v2 — Parses YAML fragments embedded in certain plugin configurations.

Configuration-Driven Generation

The config.json file acts as the declarative blueprint, separating data sources from output targets without requiring code changes.

Input Configuration

The input array defines data sources using type, action (add/remove), and args fields:

• MaxMind CSV — maxmindGeoLite2CountryCSV type reads GeoLite2-Country-Locations-en.csv, GeoLite2-Country-Blocks-IPv4.csv, and GeoLite2-Country-Blocks-IPv6.csv.

• Remote Text Lists — text type fetches plain-text IP lists from URIs like https://raw.githubusercontent.com/17mon/china_ip_list/master/china_ip_list.txt.

• JSON Endpoints — json type extracts CIDR ranges from remote JSON APIs using gjson selectors.

• Private Ranges — private type injects standard private IP ranges (RFC 1918) into the container.

Output Configuration

The output array specifies destination formats using the same type/action/args pattern:

• V2Ray DAT — v2rayGeoIPDat type generates geoip.dat protobuf files compatible with V2Ray and Xray cores.

• MaxMind MMDB — maxmindMMDB type produces Country.mmdb files, supporting sourceMMDBURI for template inheritance and overwriteList for selective updates.

• Proxy Tool Formats — Additional types generate Clash Rule Sets, Surge rule files, and sing-box SRS databases.

Running the Pipeline Locally

Developers can replicate the weekly generation process on local machines for testing or custom builds.

Building and Executing

# Compile the binary

go build -o geoip .

# Run conversion with default configuration

./geoip convert -c ./config.json

Adding Custom Data Sources

To include a new IP list in the next generation cycle, add an entry to the input array in config.json:

{
  "type": "text",
  "action": "add",
  "args": {
    "name": "custom",
    "uri": "https://example.com/custom-ip-list.txt",
    "onlyIPType": "ipv4"
  }
}

Defining Additional Outputs

To generate a specialized V2Ray database containing only specific categories, extend the output array:

{
  "type": "v2rayGeoIPDat",
  "action": "output",
  "args": {
    "outputDir": "./output",
    "outputName": "geoip-custom.dat",
    "wantedList": ["cn", "tor", "custom"]
  }
}

Summary

• The loyalsoldier/geoip repository triggers automated builds every Thursday via cron: "0 0 * * 4" in .github/workflows/build.yml.
• A Go 1.25 binary processes data through a plugin architecture implementing InputConverter and OutputConverter interfaces.
• The pipeline downloads fresh MaxMind GeoLite2-Country and GeoLite2-ASN data using authenticated API requests.
• Core data structures in lib/container.go manage CIDR additions and removals across all input sources.
• Output formats include V2Ray DAT, MaxMind MMDB, Clash rules, and sing-box SRS via specialized dependencies like mmdbwriter and protobuf.
• The entire workflow is configurable through config.json, requiring no code changes to modify inputs or outputs.

Frequently Asked Questions

How does the repository schedule its weekly GeoIP generation?

The repository uses a GitHub Actions workflow defined in .github/workflows/build.yml that triggers automatically via the cron expression 0 0 * * 4, which executes at 00:00 UTC every Thursday without requiring manual intervention.

What validates the integrity of generated MaxMind MMDB files?

After conversion, the workflow installs the github.com/maxmind/mmdbverify tool and validates each generated .mmdb file against MaxMind's official specification before including it in the GitHub Release, ensuring database compatibility with standard readers.

Which Go version and dependencies power the conversion engine?

The pipeline requires Go 1.25 as specified in go.mod, with critical dependencies including github.com/maxmind/mmdbwriter for MMDB creation, github.com/klauspost/compress for archive handling, and google.golang.org/protobuf for V2Ray DAT serialization.

How can I customize the weekly output formats without modifying Go code?

Edit the config.json file at the repository root to add new entries to the input array (for data sources) or output array (for target formats), then commit the changes; the next scheduled run will automatically pick up the new configuration and generate the additional files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}