How the vt_cluster Plugin Enables Parallel Test Execution Across Multiple Hosts

The vt_cluster plugin orchestrates parallel test execution across multiple hosts by using ThreadPoolExecutor to start and stop agent servers on all cluster nodes concurrently during the Avocado job lifecycle, while delegating actual test parallelism to the pre-configured remote agents rather than the Avocado runner itself.

The vt_cluster plugin in the avocado-framework/avocado-vt repository transforms single-host Avocado-VT jobs into distributed testing operations. By hooking into the job lifecycle and managing remote agent processes in parallel, this plugin enables seamless parallel test execution across multiple hosts without requiring the Avocado runner itself to handle the concurrency.

Architecture of the vt_cluster Plugin

The VTCluster plugin (registered as vt-cluster) serves as the entry point that makes a virtual test cluster usable by Avocado-VT tests. Its architecture relies on four core components that coordinate distributed resources.

Core Components

• VTCluster (located in avocado_vt/plugins/vt_cluster.py): The main plugin class that hooks into the Avocado job lifecycle via pre_tests and post_tests methods. It manages the concurrent lifecycle of agent servers across all cluster nodes and handles the central resource manager initialization.

• _Cluster (located in virttest/vt_cluster/__init__.py): A singleton that holds the description of all nodes (IP/hostname, tags, credentials) loaded from cluster metadata. It provides the get_all_nodes() method that returns Node objects for iteration.

• Node (located in virttest/vt_cluster/node.py): Represents a single remote host. Implements start_agent_server() and stop_agent_server() methods that the plugin calls to establish communication channels with each host.

• resmgr (located in virttest/vt_resmgr/resource_manager.py): The cluster-wide resource manager that runs VT-specific background services. Started once before the agents via _setup_mgr(), it coordinates shared resources across the distributed environment.

Parallelization Mechanics

The plugin implements true parallelism during environment preparation and teardown, ensuring all remote hosts are ready before test execution begins.

Concurrent Node Setup

The _setup_nodes() method in avocado_vt/plugins/vt_cluster.py creates a ThreadPoolExecutor sized to the number of cluster nodes. It submits node.start_agent_server() for each host concurrently, then monitors completion using as_completed(). If any node fails to start its agent, the plugin raises a ClusterSetupError immediately.

# From avocado_vt/plugins/vt_cluster.py

def _setup_nodes(self):
    """Starts agent servers on all cluster nodes in parallel."""
    nodes = cluster.get_all_nodes()
    if not nodes:
        return

    def __start_node_agent(node):
        try:
            node.start_agent_server()
            return node, None
        except Exception as err:
            return node, err

    with ThreadPoolExecutor(max_workers=len(nodes)) as executor:
        future_to_node = {
            executor.submit(__start_node_agent, node): node for node in nodes
        }
        for future in as_completed(future_to_node):
            node, error = future.result()
            if error:
                raise ClusterSetupError(
                    f"Failed to start agent on node '{node.name}': {error}"
                )

Concurrent Teardown and Log Collection

The _cleanup_nodes() method mirrors the setup logic. It copies remote agent logs back to the job log directory and stops each agent server, all within a ThreadPoolExecutor. This ensures rapid cleanup even when managing dozens of hosts.

# From avocado_vt/plugins/vt_cluster.py

def _cleanup_nodes(self, job):
    """Collect logs and stop agents on all nodes in parallel."""
    cluster_dir = os.path.join(job.logdir, "cluster")
    nodes = cluster.get_all_nodes()

    def __cleanup_agent_node(node):
        node_dir = os.path.join(cluster_dir, node.name)
        os.makedirs(node_dir, exist_ok=True)
        upload_error = stop_error = None
        
        try:
            remote_path = node.proxy.core.get_agent_log_filename()
            if remote_path:
                node.copy_files_from(node_dir, remote_path)
        except Exception as err:
            upload_error = err
        finally:
            try:
                node.stop_agent_server()
            except Exception as stop_err:
                stop_error = ClusterCleanupError(stop_err)
        return node, upload_error, stop_error

    with ThreadPoolExecutor(max_workers=len(nodes)) as executor:
        futures = {executor.submit(__cleanup_agent_node, n): n for n in nodes}
        for future in as_completed(futures):
            node, upload_error, stop_error = future.result()
            # Error handling and logging...

Integration with the Avocado Test Runner

The VT runner explicitly disables Avocado-level parallelism to prevent conflicts with the cluster-managed concurrency. In avocado_vt/plugins/vt_runner.py, the VTTestRunner enforces run.max_parallel_tasks = 1, ensuring that the Avocado runner executes only one test instance at a time. The parallelism is delegated to the cluster layer: the single test instance can drive multiple remote VMs concurrently through the already-running agents on each node.

End-to-End Execution Flow

The plugin hooks into specific Avocado job lifecycle points to prepare the distributed environment:

1. Job Start: Avocado instantiates the VTCluster plugin and loads cluster metadata from cluster_metadata.json via virttest.vt_cluster._Cluster.

2. pre_tests Hook:

   • _setup_nodes() starts agent servers on every host in parallel
   • _setup_mgr() initializes the central resource manager
   • node_properties.save_properties() persists node-specific metadata

3. Test Execution: The test code accesses prepared nodes through the cluster singleton. Since agents are already running, methods like node.exec() or node.copy_files_to() execute immediately without additional synchronization. Multiple VMs on different hosts operate simultaneously because each host maintains its own agent process.

4. post_tests Hook:

   • _cleanup_mgr() stops the resource manager
   • node_properties.remove_properties() clears persisted data
   • _cleanup_nodes() gathers logs and stops agents in parallel

Configuring and Using the Cluster

To enable parallel test execution across multiple hosts, you must define your cluster topology and access nodes within your test code.

Cluster Configuration File

Create a cluster.json file (typically placed under $HOME/.config/avocado/vt/cluster.json) defining your remote hosts:

{
  "nodes": [
    {
      "name": "node01",
      "address": "192.168.122.101",
      "username": "root",
      "password": "redhat",
      "tag": "primary"
    },
    {
      "name": "node02",
      "address": "192.168.122.102",
      "username": "root",
      "password": "redhat",
      "tag": "secondary"
    }
  ]
}

When the job starts, VTCluster reads this configuration via virttest.vt_cluster._Cluster and launches agents on both node01 and node02 concurrently.

Accessing Nodes in Test Code

Within your VT test, access the prepared nodes through the cluster singleton to execute operations across hosts:

def test_multi_host(mytest):
    from virttest.vt_cluster import cluster
    
    # Retrieve nodes by their configured tags

    node_a = cluster.get_node_by_tag('primary')
    node_b = cluster.get_node_by_tag('secondary')
    
    # Create and start VMs on different hosts simultaneously

    vm_a = mytest.create_vm(params={'vm_name': 'vmA'}, node=node_a)
    vm_b = mytest.create_vm(params={'vm_name': 'vmB'}, node=node_b)
    
    vm_a.start()
    vm_b.start()
    
    # Perform distributed testing operations...

No additional threading is required in the test code; the agents on remote hosts handle command execution concurrently.

Summary

• The vt_cluster plugin enables parallel test execution across multiple hosts by managing remote agent lifecycles concurrently using ThreadPoolExecutor.
• Parallel setup: The _setup_nodes() method in avocado_vt/plugins/vt_cluster.py starts all agent servers simultaneously before test execution begins.
• Parallel teardown: The _cleanup_nodes() method collects logs and stops agents in parallel during job cleanup.
• Architecture: The _Cluster singleton (virttest/vt_cluster/__init__.py) stores node definitions, while individual Node objects (virttest/vt_cluster/node.py) handle the actual agent process management.
• Runner integration: The VT runner disables Avocado-level parallelism (max_parallel_tasks=1) to prevent conflicts, delegating concurrency to the cluster layer.
• Usage: Tests access running agents through the cluster API to execute commands on multiple hosts simultaneously without manual synchronization.

Frequently Asked Questions

What is the primary role of the VTCluster plugin?

The VTCluster plugin acts as a job lifecycle hook that prepares and tears down distributed test environments. According to the source code in avocado_vt/plugins/vt_cluster.py, it starts agent servers on all configured remote nodes in parallel during the pre_tests phase and stops them during post_tests, ensuring a fully functional multi-host environment exists before any test code executes.

How does the plugin handle failures during agent startup?

The _setup_nodes() method uses ThreadPoolExecutor with as_completed() to monitor each node's agent startup. If any node raises an exception during start_agent_server(), the plugin immediately raises a ClusterSetupError with the specific node name and error details, failing fast before tests begin. This prevents partial cluster states where only some nodes are operational.

Why must Avocado-level parallelism be disabled when using vt_cluster?

The VTTestRunner in avocado_vt/plugins/vt_runner.py explicitly sets run.max_parallel_tasks = 1 because the parallelism is delegated to the cluster layer rather than the Avocado runner. The runner executes a single test instance that internally coordinates multiple remote hosts through the pre-started agents. Enabling Avocado-level parallelism would create resource conflicts and synchronization issues with the cluster-managed remote processes.

Where is the cluster node metadata stored?

Node metadata is stored in cluster_metadata.json (typically located under $HOME/.config/avocado/vt/). The _Cluster singleton in virttest/vt_cluster/__init__.py loads this file during plugin initialization to populate Node objects with IP addresses, credentials, and tags. The plugin also persists runtime node properties using node_properties.save_properties() during the setup phase.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}