How to Implement Automated Testing for Custom GeoIP Data Transformations and Format Conversions

To implement automated testing for custom GeoIP data transformations in the loyalsoldier/geoip repository, create Go test files that instantiate a lib.Container, register input and output converters via the library's registration functions, feed deterministic test fixtures through the conversion pipeline, and assert the output matches expected CIDR lists or format structures.

The loyalsoldier/geoip repository is a Go-based command-line tool that ingests GeoIP data from multiple sources, processes it through a pipeline of transformations, and exports it to various output formats like V2Ray, MaxMind, and Clash rule sets. Because the project uses a plugin-based architecture where each input and output format is implemented as a registered converter, automated testing ensures that custom transformations correctly handle CIDR merging, filtering, and format-specific serialization without regressions.

Understanding the GeoIP Plugin Architecture

Before writing tests, you must understand how the conversion pipeline operates. The core data structure is the lib.Container, defined in lib/container.go, which holds all IP entries grouped by list name (e.g., geoip:cn). Input converters read raw data and populate this container, while output converters serialize the container to a specific format.

Each plugin registers itself during initialization using lib.RegisterInputConfigCreator or lib.RegisterOutputConfigCreator, typically inside an init() function. For example, plugin/v2ray/dat_in.go registers the v2rayGeoIPDatIn input type, while plugin/v2ray/dat_out.go registers the corresponding output converter. Your tests must invoke these same registration mechanisms to ensure you are testing the actual code paths used by the CLI.

Setting Up Your Test Environment

Create test files adjacent to the plugin code you are validating, following Go conventions. Name the file *_test.go so the go test runner recognizes it automatically.

For example, to test V2Ray format conversions, add:

package v2ray

import (
    "os"
    "testing"

    "github.com/Loyalsoldier/geoip/lib"
)

Store deterministic test fixtures in a testdata/ subdirectory within your plugin folder. This directory is treated specially by Go tooling and is preserved when running tests. Place minimal .dat, .mmdb, or JSON files here that contain known CIDRs (such as 1.2.3.0/24 and 2.2.2.2/32) to serve as controlled inputs.

Building a Test Pipeline

A robust test exercises the full pipeline: input conversion, container processing, and output conversion. You need helper functions to instantiate the container and load registered converters.

Initializing the Container

Always start with a fresh container to avoid state pollution between tests:

func newTestContainer(t *testing.T) lib.Container {
    c, err := lib.NewContainer()
    if err != nil {
        t.Fatalf("failed to create container: %v", err)
    }
    return c
}

The lib.NewContainer() function returns an empty container with default configuration, ready to receive entries from any input converter.

Loading Input Converters

Use the library's registration lookup to instantiate the exact converter the CLI would use:

func loadInput(t *testing.T, typ string, action lib.Action, data []byte) lib.InputConverter {
    creator, ok := lib.GetInputConfigCreator(typ)
    if !ok {
        t.Fatalf("input type %q not registered", typ)
    }
    conv, err := creator(action, data)
    if err != nil {
        t.Fatalf("failed to create input converter: %v", err)
    }
    return conv
}

This pattern ensures you are testing the actual initialization logic found in files like plugin/maxmind/maxmind_country_csv_in.go or plugin/plaintext/json_in.go, not just mock implementations.

Loading Output Converters

Similarly, obtain output converters using lib.GetOutputConfigCreator:

func loadOutput(t *testing.T, typ string, action lib.Action, data []byte) lib.OutputConverter {
    creator, ok := lib.GetOutputConfigCreator(typ)
    if !ok {
        t.Fatalf("output type %q not registered", typ)
    }
    conv, err := creator(action, data)
    if err != nil {
        t.Fatalf("failed to create output converter: %v", err)
    }
    return conv
}

Writing an End-to-End Conversion Test

Combine these helpers to test a complete transformation. The following example validates that a custom V2Ray .dat file correctly converts to plaintext CIDR lists:

func TestV2RayDatConversion(t *testing.T) {
    // 1. Prepare container
    container := newTestContainer(t)

    // 2. Load input referencing a fixture in testdata/
    input := loadInput(t, "v2rayGeoIPDatIn", lib.ActionAdd, []byte(`{"source":"testdata/custom.dat"}`))

    // 3. Feed the input to the container
    var err error
    container, err = input.Input(container)
    if err != nil {
        t.Fatalf("input conversion failed: %v", err)
    }

    // 4. Configure output to write to a temporary file
    tmpFile := t.TempDir() + "/out.txt"
    outConfig := []byte(`{"path":"` + tmpFile + `"}`)
    
    output := loadOutput(t, "textOut", lib.ActionAdd, outConfig)

    // 5. Run the output conversion
    if err = output.Output(container); err != nil {
        t.Fatalf("output conversion failed: %v", err)
    }

    // 6. Verify the resulting CIDRs
    data, err := os.ReadFile(tmpFile)
    if err != nil {
        t.Fatalf("cannot read output file: %v", err)
    }
    expected := "1.2.3.0/24\n2.2.2.2/32\n"
    if string(data) != expected {
        t.Fatalf("unexpected output.\nGot:\n%s\nWant:\n%s", string(data), expected)
    }
}

Use t.TempDir() to ensure the test writes to an isolated, auto-cleaned directory. The expected output string should reflect the exact CIDRs embedded in your testdata/custom.dat fixture.

Managing Test Fixtures

Generate minimal fixtures to keep tests fast and deterministic. For binary formats like V2Ray's .dat or MaxMind's .mmdb, you can create small reference files using the CLI itself:

go run ./ convert -c testdata/config.json

Where config.json references a tiny text list containing only the CIDRs you need to test. Commit these generated files to plugin/<name>/testdata/ so the test suite remains self-contained and works offline.

The reference implementation in plugin/special/test.go demonstrates how to inject a known CIDR (127.0.0.0/8) programmatically without external files, which you can adapt for unit-style tests that don't require full file I/O.

Integrating with CI/CD

The repository's existing continuous integration pipeline, defined in .github/workflows/build.yml, executes go test ./... across all packages. By placing your *_test.go files alongside the plugin code and including fixtures in testdata/, your automated tests for custom GeoIP data transformations automatically integrate with the CI workflow without additional configuration.

Run tests locally during development to validate your changes:

go test ./...

Or target a specific package for faster iteration:

go test ./plugin/v2ray/...

Troubleshooting Common Testing Issues

When implementing automated testing for format conversions, you may encounter these specific failures:

• Nil container errors: Always initialize the container with lib.NewContainer() before passing it to Input(). Attempting to use a nil container causes panics during entry insertion.

• Missing registration errors: If lib.GetInputConfigCreator returns ok = false, verify that the plugin's init() function actually calls the registration function. Check files like plugin/v2ray/dat_in.go for the correct registration pattern.

• Path resolution failures: Output plugins like textOut often require an absolute or relative path in their JSON configuration. Use t.TempDir() to generate a writable temporary path and inject it into the configuration bytes, as shown in the test example above.

• Fixture bloat: Large MaxMind or V2Ray databases slow down tests and consume memory. Keep fixtures under 10 KB, containing only IPv4 and IPv6 edge cases (e.g., /32, /128, overlapping ranges) rather than full commercial databases.

Extending Tests to Other Formats

The same testing pattern applies across all supported formats. Simply change the type strings and configuration JSON to target different converters:

• MaxMind CSV input: Use maxmindGeoLite2CountryCSVIn from plugin/maxmind/maxmind_country_csv_in.go
• JSON plaintext input: Use jsonIn from plugin/plaintext/json_in.go
• MaxMind MMDB output: Use maxmindMMDB from plugin/maxmind/maxmind_country_mmdb_out.go
• Clash rule set output: Use clashRuleSet from plugin/plaintext/clash_out.go

Each converter expects specific configuration fields documented in configuration.md. Pass these as JSON bytes to the creator functions in your tests.

Summary

• Create *_test.go files next to the plugin code you are validating to ensure the Go test runner discovers them automatically.
• Initialize a fresh lib.Container using lib.NewContainer() for every test to prevent cross-test contamination.
• Load converters via lib.GetInputConfigCreator and lib.GetOutputConfigCreator to exercise the actual registration logic used by the CLI.
• Store minimal binary and text fixtures in a testdata/ subdirectory, generating them with the CLI if necessary.
• Capture output to temporary directories using t.TempDir() and assert against expected CIDR strings or format-specific structures.
• Execute go test ./... to validate your automated tests locally; the existing .github/workflows/build.yml CI pipeline will automatically run these tests on every pull request.

Frequently Asked Questions

How do I test a custom input plugin that isn't registered in the main library?

Ensure your plugin file includes an init() function that calls lib.RegisterInputConfigCreator with a unique type string. Without this registration step, lib.GetInputConfigCreator cannot locate your converter, and the test will fail with a "not registered" error. See plugin/special/test.go for a minimal working example of registration.

What is the best way to handle binary format fixtures like .mmdb or .dat files?

Generate small, deterministic fixtures using the CLI tool itself (e.g., go run ./ convert with a minimal config), then commit these files to your testdata/ directory. Keep them under 10 KB by including only a handful of CIDRs that exercise IPv4, IPv6, and edge cases like single IPs (/32) and network ranges.

Can I run tests in parallel for faster CI execution?

Yes, but use caution when writing to files. Either avoid t.Parallel() for tests that write to the filesystem, or ensure each parallel test uses a unique subdirectory within t.TempDir(). The lib.Container itself is safe for concurrent use, but output converters that write to fixed paths may cause race conditions.

How do I verify that my transformation correctly filters or merges CIDRs?

Populate the input fixture with overlapping or adjacent CIDRs (e.g., 192.168.0.0/24 and 192.168.1.0/24), run them through the container, and inspect the output. The lib.Container automatically merges entries when possible. Assert that the output contains the expected merged ranges or that excluded ranges are absent, depending on your filter logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}