How to Migrate from Other Elasticsearch Management Tools to INFINI Console

INFINI Console provides a comprehensive Data Migration workflow that transfers indices, mappings, settings, and documents from any external Elasticsearch or OpenSearch cluster through a guided UI wizard and REST API endpoints, featuring partition-based parallelism and incremental detection for zero-downtime migrations.

The infinilabs/console repository includes a robust migration engine designed to streamline transitions from third-party Elasticsearch management platforms. This system orchestrates complex data movements through a multi-layered architecture that handles everything from cluster authentication to bulk reindexing operations while maintaining full audit trails.

Migration Architecture Overview

The migration process operates across three distinct architectural layers, each defined in specific source files within the repository.

UI and Orchestration Layer

The Data Tools section in the left navigation guides users through a wizard that creates a migration task. According to docs/content.en/docs/reference/migration/migration.md, this interface collects source and target cluster IDs, index selections, partitioning configurations, and incremental settings before triggering the backend job.

Backend Service Layer

A set of HTTP handlers exposes a REST API under /migration/data. As implemented in web/mock/migration/migration.js, these handlers validate requests, persist task definitions, and hand jobs to the migration engine. The mock handlers illustrate the production API contract for creating, querying, and updating migration tasks.

Migration Engine Layer

The engine runs as a long-lived background worker defined in service/alerting/engine.go. It reads task definitions from modules/elastic/api/view.go where the pipeline is configured with pipeline_id: "cluster_migration". The engine executes Elasticsearch bulk-reindex operations, splitting indices into independent partitions to isolate failures and enable parallel processing.

Step-by-Step Migration Workflow

Follow this 11-step process to move data from external clusters into INFINI Console management:

1. Open the Data Migration wizard by clicking Data Tools → Migration in the left menu, then press New to initialize a task.

2. Select source and destination clusters. Choose any external Elasticsearch 7.x/8.x or OpenSearch cluster as the source, and select the INFINI Console managed cluster as the target.

3. Choose indices to migrate by clicking Select Migration Indices. For pre-7.x indices containing multiple types, the UI automatically splits each type into a separate target index.

4. Configure mappings and settings by reviewing the source mappings displayed on the left panel. Copy them to the target side for editing, or enable Auto Optimize to allow automatic compatibility optimizations such as ILM updates.

5. Define data range and partitioning. Leave the data range empty for full migration, or specify a time window. Enable partitioning by selecting a date or numeric field with a step value (e.g., 5m). The engine creates one sub-task per partition, improving fault tolerance and parallelism.

6. Configure runtime parameters including execution nodes (gateway instances), bulk size, parallelism, and scroll timeout. Default values work for most scenarios unless specific performance tuning is required.

7. Enable incremental migration for continuously written indices such as logs or metrics. Select Detect Incremental Data, provide an incremental field (typically a timestamp), and set a write delay (default 15 minutes) to avoid missing in-flight documents.

8. Create and start the task by pressing Create Task. The task appears in the Migration task list where you can click Start to launch the job.

9. Monitor progress through the task Details page. The UI displays a grid of colored squares representing sub-tasks: green indicates completion, gray indicates pending status, and red indicates errors.

10. Handle errors by clicking red squares to view detailed logs showing exact failures such as mapping conflicts or bulk reindex errors. Because each partition operates independently, you can retry only the failed sub-task without re-processing the entire index.

11. Finalize the migration once all squares display green. Optionally run a Data Comparison task to validate parity between source and target clusters.

Programmatic Migration via REST API

For automated or scripted migrations, interact directly with the REST endpoints defined in web/mock/migration/migration.js.

Creating a Migration Task

Submit a POST request to /migration/data with a payload defining the pipeline configuration:

POST /migration/data HTTP/1.1
Content-Type: application/json

{
  "pipeline": {
    "id": "cluster_migration",
    "config": {
      "cluster": {
        "source": { "id": "src-cluster-id", "name": "es-source" },
        "target": { "id": "dst-cluster-id", "name": "es-destination" }
      },
      "indices": [
        {
          "id": "test-index",
          "raw_filter": {
            "range": { "timestamp": { "gte": "2022-01-01", "lte": "2022-12-31" } }
          },
          "partition": {
            "field_name": "timestamp",
            "field_type": "date",
            "step": "5m"
          },
          "source": { "name": "test", "docs": 1800000 },
          "target": { "name": "test_bak", "docs": 0 }
        }
      ],
      "settings": {
        "bulk_size": { "documents": 1000, "store_size_in_mb": 20 },
        "parallel_indices": 2,
        "parallel_task_per_index": 1,
        "scroll_size": { "documents": 1000, "timeout": "5m" }
      },
      "creator": { "id": "10000", "name": "admin" }
    }
  }
}

Querying Task Status

Retrieve current status and pipeline configuration:

GET /migration/data/:id HTTP/1.1
Accept: application/json

The response includes the task status and full configuration:

{
  "found": true,
  "_id": "c97um9tath2fgbc3jbxx",
  "_source": {
    "status": "running",
    "creator": { "name": "admin", "id": "10000" },
    "pipeline": { }
  }
}

Enabling Incremental Detection

Toggle incremental mode for live indices:

POST /migration/data/:id/_status HTTP/1.1
Content-Type: application/json

{
  "action": "detect_incremental",
  "incremental_field": "timestamp",
  "write_delay": "15m",
  "detect_interval": "15m"
}

This updates pipeline.config.incremental and initiates periodic detection jobs.

Security and Audit Controls

The migration system enforces strict access controls and maintains comprehensive audit trails. The permission constants DataMigrationRead and DataMigrationAll defined in core/security/enum/const.go gate access to migration functionality. Every operation generates an audit log entry with ResourceTypeDataMigration as defined in model/audit_log.go, and can trigger notifications of type MessageTypeMigration per model/notification.go.

Summary

• INFINI Console migrates data through a Data Migration workflow using the cluster_migration pipeline ID.
• The system supports partition-based parallelism that isolates failures to individual sub-tasks.
• Incremental detection enables zero-downtime migrations for actively written indices.
• REST API endpoints under /migration/data allow full programmatic control.
• Security is enforced via DataMigrationRead and DataMigrationAll permissions with full audit logging.

Frequently Asked Questions

Can I migrate from Elasticsearch 7.x and 8.x to INFINI Console?

Yes. The migration engine supports any external Elasticsearch 7.x, 8.x, or OpenSearch cluster that the Console can reach via network connection. The UI wizard automatically handles version-specific mapping conversions, including splitting pre-7.x multi-type indices into separate target indices.

How does partition-based migration improve reliability?

Partitioning splits large indices into smaller sub-tasks based on date or numeric field ranges. As implemented in the migration engine, each partition operates as an independent bulk-reindex operation. If one partition encounters a mapping conflict or network error, only that specific sub-task fails and requires retry, rather than restarting the entire index migration.

What happens if a migration sub-task fails?

Failed sub-tasks appear as red squares in the monitoring grid. Clicking the square reveals detailed error logs showing the specific failure reason, such as bulk reindex errors or mapping conflicts. Because partitions are independent, you can retry individual failed sub-tasks through the UI or API without re-processing successfully completed portions of the index.

Does INFINI Console support incremental migration for live indices?

Yes. Enable Detect Incremental Data during task creation or via the POST /migration/data/:id/_status endpoint with action: "detect_incremental". Provide a timestamp field and write delay (typically 15 minutes) to allow in-flight documents to settle. The engine periodically polls for new documents and copies them to the target cluster without requiring a full re-migration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}