# How to Migrate from Other Elasticsearch Management Tools to INFINI Console

> Seamlessly migrate Elasticsearch data to INFINI Console. Our comprehensive workflow transfers indices, mappings, and documents with partition-based parallelism and zero-downtime support.

- Repository: [INFINI Labs/console](https://github.com/infinilabs/console)
- Tags: migration-guide
- Published: 2026-03-04

---

**INFINI Console provides a comprehensive Data Migration workflow that transfers indices, mappings, settings, and documents from any external Elasticsearch or OpenSearch cluster through a guided UI wizard and REST API endpoints, featuring partition-based parallelism and incremental detection for zero-downtime migrations.**

The `infinilabs/console` repository includes a robust migration engine designed to streamline transitions from third-party Elasticsearch management platforms. This system orchestrates complex data movements through a multi-layered architecture that handles everything from cluster authentication to bulk reindexing operations while maintaining full audit trails.

## Migration Architecture Overview

The migration process operates across three distinct architectural layers, each defined in specific source files within the repository.

### UI and Orchestration Layer

The **Data Tools** section in the left navigation guides users through a wizard that creates a **migration task**. According to [`docs/content.en/docs/reference/migration/migration.md`](https://github.com/infinilabs/console/blob/main/docs/content.en/docs/reference/migration/migration.md), this interface collects source and target cluster IDs, index selections, partitioning configurations, and incremental settings before triggering the backend job.

### Backend Service Layer

A set of HTTP handlers exposes a REST API under `/migration/data`. As implemented in [`web/mock/migration/migration.js`](https://github.com/infinilabs/console/blob/main/web/mock/migration/migration.js), these handlers validate requests, persist task definitions, and hand jobs to the migration engine. The mock handlers illustrate the production API contract for creating, querying, and updating migration tasks.

### Migration Engine Layer

The engine runs as a long-lived background worker defined in [`service/alerting/engine.go`](https://github.com/infinilabs/console/blob/main/service/alerting/engine.go). It reads task definitions from [`modules/elastic/api/view.go`](https://github.com/infinilabs/console/blob/main/modules/elastic/api/view.go) where the pipeline is configured with `pipeline_id: "cluster_migration"`. The engine executes Elasticsearch bulk-reindex operations, splitting indices into independent partitions to isolate failures and enable parallel processing.

## Step-by-Step Migration Workflow

Follow this 11-step process to move data from external clusters into INFINI Console management:

1. **Open the Data Migration wizard** by clicking **Data Tools → Migration** in the left menu, then press **New** to initialize a task.

2. **Select source and destination clusters**. Choose any external Elasticsearch 7.x/8.x or OpenSearch cluster as the source, and select the INFINI Console managed cluster as the target.

3. **Choose indices to migrate** by clicking **Select Migration Indices**. For pre-7.x indices containing multiple types, the UI automatically splits each type into a separate target index.

4. **Configure mappings and settings** by reviewing the source mappings displayed on the left panel. Copy them to the target side for editing, or enable **Auto Optimize** to allow automatic compatibility optimizations such as ILM updates.

5. **Define data range and partitioning**. Leave the data range empty for full migration, or specify a time window. Enable partitioning by selecting a **date** or **numeric** field with a step value (e.g., `5m`). The engine creates one sub-task per partition, improving fault tolerance and parallelism.

6. **Configure runtime parameters** including execution nodes (gateway instances), bulk size, parallelism, and scroll timeout. Default values work for most scenarios unless specific performance tuning is required.

7. **Enable incremental migration** for continuously written indices such as logs or metrics. Select **Detect Incremental Data**, provide an incremental field (typically a timestamp), and set a write delay (default 15 minutes) to avoid missing in-flight documents.

8. **Create and start the task** by pressing **Create Task**. The task appears in the **Migration task list** where you can click **Start** to launch the job.

9. **Monitor progress** through the task **Details** page. The UI displays a grid of colored squares representing sub-tasks: **green** indicates completion, **gray** indicates pending status, and **red** indicates errors.

10. **Handle errors** by clicking red squares to view detailed logs showing exact failures such as mapping conflicts or bulk reindex errors. Because each partition operates independently, you can retry only the failed sub-task without re-processing the entire index.

11. **Finalize the migration** once all squares display green. Optionally run a **Data Comparison** task to validate parity between source and target clusters.

## Programmatic Migration via REST API

For automated or scripted migrations, interact directly with the REST endpoints defined in [`web/mock/migration/migration.js`](https://github.com/infinilabs/console/blob/main/web/mock/migration/migration.js).

### Creating a Migration Task

Submit a POST request to `/migration/data` with a payload defining the pipeline configuration:

```http
POST /migration/data HTTP/1.1
Content-Type: application/json

{
  "pipeline": {
    "id": "cluster_migration",
    "config": {
      "cluster": {
        "source": { "id": "src-cluster-id", "name": "es-source" },
        "target": { "id": "dst-cluster-id", "name": "es-destination" }
      },
      "indices": [
        {
          "id": "test-index",
          "raw_filter": {
            "range": { "timestamp": { "gte": "2022-01-01", "lte": "2022-12-31" } }
          },
          "partition": {
            "field_name": "timestamp",
            "field_type": "date",
            "step": "5m"
          },
          "source": { "name": "test", "docs": 1800000 },
          "target": { "name": "test_bak", "docs": 0 }
        }
      ],
      "settings": {
        "bulk_size": { "documents": 1000, "store_size_in_mb": 20 },
        "parallel_indices": 2,
        "parallel_task_per_index": 1,
        "scroll_size": { "documents": 1000, "timeout": "5m" }
      },
      "creator": { "id": "10000", "name": "admin" }
    }
  }
}

```

### Querying Task Status

Retrieve current status and pipeline configuration:

```http
GET /migration/data/:id HTTP/1.1
Accept: application/json

```

The response includes the task status and full configuration:

```json
{
  "found": true,
  "_id": "c97um9tath2fgbc3jbxx",
  "_source": {
    "status": "running",
    "creator": { "name": "admin", "id": "10000" },
    "pipeline": { }
  }
}

```

### Enabling Incremental Detection

Toggle incremental mode for live indices:

```http
POST /migration/data/:id/_status HTTP/1.1
Content-Type: application/json

{
  "action": "detect_incremental",
  "incremental_field": "timestamp",
  "write_delay": "15m",
  "detect_interval": "15m"
}

```

This updates `pipeline.config.incremental` and initiates periodic detection jobs.

## Security and Audit Controls

The migration system enforces strict access controls and maintains comprehensive audit trails. The permission constants `DataMigrationRead` and `DataMigrationAll` defined in [`core/security/enum/const.go`](https://github.com/infinilabs/console/blob/main/core/security/enum/const.go) gate access to migration functionality. Every operation generates an audit log entry with `ResourceTypeDataMigration` as defined in [`model/audit_log.go`](https://github.com/infinilabs/console/blob/main/model/audit_log.go), and can trigger notifications of type `MessageTypeMigration` per [`model/notification.go`](https://github.com/infinilabs/console/blob/main/model/notification.go).

## Summary

- INFINI Console migrates data through a **Data Migration** workflow using the `cluster_migration` pipeline ID.
- The system supports **partition-based parallelism** that isolates failures to individual sub-tasks.
- **Incremental detection** enables zero-downtime migrations for actively written indices.
- REST API endpoints under `/migration/data` allow full programmatic control.
- Security is enforced via `DataMigrationRead` and `DataMigrationAll` permissions with full audit logging.

## Frequently Asked Questions

### Can I migrate from Elasticsearch 7.x and 8.x to INFINI Console?

Yes. The migration engine supports any external Elasticsearch 7.x, 8.x, or OpenSearch cluster that the Console can reach via network connection. The UI wizard automatically handles version-specific mapping conversions, including splitting pre-7.x multi-type indices into separate target indices.

### How does partition-based migration improve reliability?

Partitioning splits large indices into smaller sub-tasks based on date or numeric field ranges. As implemented in the migration engine, each partition operates as an independent bulk-reindex operation. If one partition encounters a mapping conflict or network error, only that specific sub-task fails and requires retry, rather than restarting the entire index migration.

### What happens if a migration sub-task fails?

Failed sub-tasks appear as red squares in the monitoring grid. Clicking the square reveals detailed error logs showing the specific failure reason, such as bulk reindex errors or mapping conflicts. Because partitions are independent, you can retry individual failed sub-tasks through the UI or API without re-processing successfully completed portions of the index.

### Does INFINI Console support incremental migration for live indices?

Yes. Enable **Detect Incremental Data** during task creation or via the `POST /migration/data/:id/_status` endpoint with `action: "detect_incremental"`. Provide a timestamp field and write delay (typically 15 minutes) to allow in-flight documents to settle. The engine periodically polls for new documents and copies them to the target cluster without requiring a full re-migration.