# How the vt_cluster Plugin Enables Parallel Test Execution Across Multiple Hosts

> Discover how the vt_cluster plugin enables parallel test execution across multiple hosts. Learn how it orchestrates agent servers for concurrent testing on cluster nodes.

- Repository: [avocado/avocado-vt](https://github.com/avocado-framework/avocado-vt)
- Tags: how-to-guide
- Published: 2026-02-25

---

**The vt_cluster plugin orchestrates parallel test execution across multiple hosts by using ThreadPoolExecutor to start and stop agent servers on all cluster nodes concurrently during the Avocado job lifecycle, while delegating actual test parallelism to the pre-configured remote agents rather than the Avocado runner itself.**

The `vt_cluster` plugin in the `avocado-framework/avocado-vt` repository transforms single-host Avocado-VT jobs into distributed testing operations. By hooking into the job lifecycle and managing remote agent processes in parallel, this plugin enables seamless parallel test execution across multiple hosts without requiring the Avocado runner itself to handle the concurrency.

## Architecture of the vt_cluster Plugin

The `VTCluster` plugin (registered as `vt-cluster`) serves as the entry point that makes a virtual test cluster usable by Avocado-VT tests. Its architecture relies on four core components that coordinate distributed resources.

### Core Components

- **`VTCluster`** (located in [`avocado_vt/plugins/vt_cluster.py`](https://github.com/avocado-framework/avocado-vt/blob/main/avocado_vt/plugins/vt_cluster.py)): The main plugin class that hooks into the Avocado job lifecycle via `pre_tests` and `post_tests` methods. It manages the concurrent lifecycle of agent servers across all cluster nodes and handles the central resource manager initialization.

- **`_Cluster`** (located in [`virttest/vt_cluster/__init__.py`](https://github.com/avocado-framework/avocado-vt/blob/main/virttest/vt_cluster/__init__.py)): A singleton that holds the description of all nodes (IP/hostname, tags, credentials) loaded from cluster metadata. It provides the `get_all_nodes()` method that returns `Node` objects for iteration.

- **`Node`** (located in [`virttest/vt_cluster/node.py`](https://github.com/avocado-framework/avocado-vt/blob/main/virttest/vt_cluster/node.py)): Represents a single remote host. Implements `start_agent_server()` and `stop_agent_server()` methods that the plugin calls to establish communication channels with each host.

- **`resmgr`** (located in [`virttest/vt_resmgr/resource_manager.py`](https://github.com/avocado-framework/avocado-vt/blob/main/virttest/vt_resmgr/resource_manager.py)): The cluster-wide resource manager that runs VT-specific background services. Started once before the agents via `_setup_mgr()`, it coordinates shared resources across the distributed environment.

## Parallelization Mechanics

The plugin implements true parallelism during environment preparation and teardown, ensuring all remote hosts are ready before test execution begins.

### Concurrent Node Setup

The `_setup_nodes()` method in [`avocado_vt/plugins/vt_cluster.py`](https://github.com/avocado-framework/avocado-vt/blob/main/avocado_vt/plugins/vt_cluster.py) creates a `ThreadPoolExecutor` sized to the number of cluster nodes. It submits `node.start_agent_server()` for each host concurrently, then monitors completion using `as_completed()`. If any node fails to start its agent, the plugin raises a `ClusterSetupError` immediately.

```python

# From avocado_vt/plugins/vt_cluster.py

def _setup_nodes(self):
    """Starts agent servers on all cluster nodes in parallel."""
    nodes = cluster.get_all_nodes()
    if not nodes:
        return

    def __start_node_agent(node):
        try:
            node.start_agent_server()
            return node, None
        except Exception as err:
            return node, err

    with ThreadPoolExecutor(max_workers=len(nodes)) as executor:
        future_to_node = {
            executor.submit(__start_node_agent, node): node for node in nodes
        }
        for future in as_completed(future_to_node):
            node, error = future.result()
            if error:
                raise ClusterSetupError(
                    f"Failed to start agent on node '{node.name}': {error}"
                )

```

### Concurrent Teardown and Log Collection

The `_cleanup_nodes()` method mirrors the setup logic. It copies remote agent logs back to the job log directory and stops each agent server, all within a `ThreadPoolExecutor`. This ensures rapid cleanup even when managing dozens of hosts.

```python

# From avocado_vt/plugins/vt_cluster.py

def _cleanup_nodes(self, job):
    """Collect logs and stop agents on all nodes in parallel."""
    cluster_dir = os.path.join(job.logdir, "cluster")
    nodes = cluster.get_all_nodes()

    def __cleanup_agent_node(node):
        node_dir = os.path.join(cluster_dir, node.name)
        os.makedirs(node_dir, exist_ok=True)
        upload_error = stop_error = None
        
        try:
            remote_path = node.proxy.core.get_agent_log_filename()
            if remote_path:
                node.copy_files_from(node_dir, remote_path)
        except Exception as err:
            upload_error = err
        finally:
            try:
                node.stop_agent_server()
            except Exception as stop_err:
                stop_error = ClusterCleanupError(stop_err)
        return node, upload_error, stop_error

    with ThreadPoolExecutor(max_workers=len(nodes)) as executor:
        futures = {executor.submit(__cleanup_agent_node, n): n for n in nodes}
        for future in as_completed(futures):
            node, upload_error, stop_error = future.result()
            # Error handling and logging...

```

## Integration with the Avocado Test Runner

The VT runner explicitly disables Avocado-level parallelism to prevent conflicts with the cluster-managed concurrency. In [`avocado_vt/plugins/vt_runner.py`](https://github.com/avocado-framework/avocado-vt/blob/main/avocado_vt/plugins/vt_runner.py), the `VTTestRunner` enforces `run.max_parallel_tasks = 1`, ensuring that the Avocado runner executes only one test instance at a time. The parallelism is delegated to the cluster layer: the single test instance can drive multiple remote VMs concurrently through the already-running agents on each node.

## End-to-End Execution Flow

The plugin hooks into specific Avocado job lifecycle points to prepare the distributed environment:

1. **Job Start**: Avocado instantiates the `VTCluster` plugin and loads cluster metadata from [`cluster_metadata.json`](https://github.com/avocado-framework/avocado-vt/blob/main/cluster_metadata.json) via `virttest.vt_cluster._Cluster`.

2. **pre_tests Hook**: 
   - `_setup_nodes()` starts agent servers on every host in parallel
   - `_setup_mgr()` initializes the central resource manager
   - `node_properties.save_properties()` persists node-specific metadata

3. **Test Execution**: The test code accesses prepared nodes through the `cluster` singleton. Since agents are already running, methods like `node.exec()` or `node.copy_files_to()` execute immediately without additional synchronization. Multiple VMs on different hosts operate simultaneously because each host maintains its own agent process.

4. **post_tests Hook**:
   - `_cleanup_mgr()` stops the resource manager
   - `node_properties.remove_properties()` clears persisted data
   - `_cleanup_nodes()` gathers logs and stops agents in parallel

## Configuring and Using the Cluster

To enable parallel test execution across multiple hosts, you must define your cluster topology and access nodes within your test code.

### Cluster Configuration File

Create a [`cluster.json`](https://github.com/avocado-framework/avocado-vt/blob/main/cluster.json) file (typically placed under `$HOME/.config/avocado/vt/cluster.json`) defining your remote hosts:

```json
{
  "nodes": [
    {
      "name": "node01",
      "address": "192.168.122.101",
      "username": "root",
      "password": "redhat",
      "tag": "primary"
    },
    {
      "name": "node02",
      "address": "192.168.122.102",
      "username": "root",
      "password": "redhat",
      "tag": "secondary"
    }
  ]
}

```

When the job starts, `VTCluster` reads this configuration via `virttest.vt_cluster._Cluster` and launches agents on both `node01` and `node02` concurrently.

### Accessing Nodes in Test Code

Within your VT test, access the prepared nodes through the cluster singleton to execute operations across hosts:

```python
def test_multi_host(mytest):
    from virttest.vt_cluster import cluster
    
    # Retrieve nodes by their configured tags

    node_a = cluster.get_node_by_tag('primary')
    node_b = cluster.get_node_by_tag('secondary')
    
    # Create and start VMs on different hosts simultaneously

    vm_a = mytest.create_vm(params={'vm_name': 'vmA'}, node=node_a)
    vm_b = mytest.create_vm(params={'vm_name': 'vmB'}, node=node_b)
    
    vm_a.start()
    vm_b.start()
    
    # Perform distributed testing operations...

```

No additional threading is required in the test code; the agents on remote hosts handle command execution concurrently.

## Summary

- The `vt_cluster` plugin enables parallel test execution across multiple hosts by managing remote agent lifecycles concurrently using `ThreadPoolExecutor`.
- **Parallel setup**: The `_setup_nodes()` method in [`avocado_vt/plugins/vt_cluster.py`](https://github.com/avocado-framework/avocado-vt/blob/main/avocado_vt/plugins/vt_cluster.py) starts all agent servers simultaneously before test execution begins.
- **Parallel teardown**: The `_cleanup_nodes()` method collects logs and stops agents in parallel during job cleanup.
- **Architecture**: The `_Cluster` singleton ([`virttest/vt_cluster/__init__.py`](https://github.com/avocado-framework/avocado-vt/blob/main/virttest/vt_cluster/__init__.py)) stores node definitions, while individual `Node` objects ([`virttest/vt_cluster/node.py`](https://github.com/avocado-framework/avocado-vt/blob/main/virttest/vt_cluster/node.py)) handle the actual agent process management.
- **Runner integration**: The VT runner disables Avocado-level parallelism (`max_parallel_tasks=1`) to prevent conflicts, delegating concurrency to the cluster layer.
- **Usage**: Tests access running agents through the cluster API to execute commands on multiple hosts simultaneously without manual synchronization.

## Frequently Asked Questions

### What is the primary role of the VTCluster plugin?

The VTCluster plugin acts as a job lifecycle hook that prepares and tears down distributed test environments. According to the source code in [`avocado_vt/plugins/vt_cluster.py`](https://github.com/avocado-framework/avocado-vt/blob/main/avocado_vt/plugins/vt_cluster.py), it starts agent servers on all configured remote nodes in parallel during the `pre_tests` phase and stops them during `post_tests`, ensuring a fully functional multi-host environment exists before any test code executes.

### How does the plugin handle failures during agent startup?

The `_setup_nodes()` method uses `ThreadPoolExecutor` with `as_completed()` to monitor each node's agent startup. If any node raises an exception during `start_agent_server()`, the plugin immediately raises a `ClusterSetupError` with the specific node name and error details, failing fast before tests begin. This prevents partial cluster states where only some nodes are operational.

### Why must Avocado-level parallelism be disabled when using vt_cluster?

The `VTTestRunner` in [`avocado_vt/plugins/vt_runner.py`](https://github.com/avocado-framework/avocado-vt/blob/main/avocado_vt/plugins/vt_runner.py) explicitly sets `run.max_parallel_tasks = 1` because the parallelism is delegated to the cluster layer rather than the Avocado runner. The runner executes a single test instance that internally coordinates multiple remote hosts through the pre-started agents. Enabling Avocado-level parallelism would create resource conflicts and synchronization issues with the cluster-managed remote processes.

### Where is the cluster node metadata stored?

Node metadata is stored in [`cluster_metadata.json`](https://github.com/avocado-framework/avocado-vt/blob/main/cluster_metadata.json) (typically located under `$HOME/.config/avocado/vt/`). The `_Cluster` singleton in [`virttest/vt_cluster/__init__.py`](https://github.com/avocado-framework/avocado-vt/blob/main/virttest/vt_cluster/__init__.py) loads this file during plugin initialization to populate `Node` objects with IP addresses, credentials, and tags. The plugin also persists runtime node properties using `node_properties.save_properties()` during the setup phase.