How the Avocado-VT vt_joblock Plugin Prevents Concurrent Test Runs from Conflicting on Shared Resources

The vtjoblock plugin enforces exclusive execution of Avocado-VT jobs by creating unique per-job lock files, verifying no other live processes hold conflicting locks, and automatically cleaning up stale files to prevent race conditions on shared virtualization resources.

The vtjoblock plugin is a critical component of the avocado-framework/avocado-vt repository designed to prevent concurrent test runs from colliding on shared resources such as VM images, QEMU port bindings, and network namespaces. By implementing a file-based locking mechanism that checks process liveness, the plugin ensures that only one job using the VirtTest class executes at a time within a given lock directory.

How the vt_joblock Plugin Works

The plugin operates as a job-level hook that activates during the pre_tests and post_tests phases of the Avocado job lifecycle. When a job starts, the plugin creates a uniquely named lock file containing the current process ID (PID). It then scans the configured lock directory for existing lock files, verifies whether the PIDs stored in those files correspond to running processes, and aborts the current job if a conflict is detected. Upon job completion or failure, the plugin removes its lock file to release the resource.

Configuration and Initialization

The plugin's behavior is configured through Avocado's settings system, defined in avocado_vt/plugins/vt_init.py. Here, the plugins.vtjoblock section registers the dir option, specifying where lock files are stored.

# avocado_vt/plugins/vt_init.py

settings.register_option(
    "plugins.vtjoblock", "dir",
    help_msg="Directory in which to write the lock file",
    default="/tmp"
)

When the VTJobLock class is instantiated in avocado_vt/plugins/vt_joblock.py, it retrieves and expands this setting:

# avocado_vt/plugins/vt_joblock.py

lock_dir = get_settings_value(
    "plugins.vtjoblock", "dir", key_type=str, default="/tmp"
)
self.lock_dir = os.path.expanduser(lock_dir)

Lock File Creation and Detection

Generating Unique Lock Files

The _create_self_lock_file() method generates a unique filename using a pattern that includes the job ID, user ID, and an 8-character random token to avoid collisions:

# avocado_vt/plugins/vt_joblock.py

pattern = "avocado-vt-joblock-%(jobid)s-%(uid)s-%(random)s.pid"
rand = "".join([random.choice(string.ascii_lowercase + string.digits)
                for i in xrange(8)])
path = os.path.join(self.lock_dir, pattern % {...})
with open(path, "w") as lockfile:
    lockfile.write("%u" % os.getpid())

Detecting Conflicts with pid_exists

The _get_lock_files() method scans the lock directory for files matching the regular expression pattern r"avocado-vt-joblock-[0-9a-f]{40}-[0-9]+-[0-9a-z]{8}\.pid".

When checking for conflicts, the plugin iterates through existing lock files (excluding its own), reads the stored PID, and uses pid_exists() from avocado.utils.process to verify if that process is still running:

# avocado_vt/plugins/vt_joblock.py

lock_pid = int(open(path, "r").read())
if pid_exists(lock_pid):
    raise OtherProcessHoldsLockError(
        f'File "{path}" acquired by PID {lock_pid}. '
    )

If a live PID is detected, the plugin raises OtherProcessHoldsLockError, preventing the current job from starting and avoiding resource conflicts.

Cleaning Up Stale Locks

If pid_exists() returns False, indicating the process that created the lock is no longer running, the plugin treats the lock as stale and removes it:

# avocado_vt/plugins/vt_joblock.py

os.unlink(path)  # remove stale lock

This cleanup prevents dead lock files from permanently blocking new jobs after crashes or unclean shutdowns.

Scope and Activation

The plugin only activates for jobs containing tests derived from the VirtTest class. In the pre_tests method, the plugin checks the test suite:

# avocado_vt/plugins/vt_joblock.py

if any(self._get_klass_or_none(tf) is VirtTest for tf in tests):
    self._lock(job)

This scope limitation ensures that non-virtualization tests or other Avocado jobs that don't use shared virtualization resources are not unnecessarily serialized.

After the test suite completes, the post_tests method removes the lock file:

# avocado_vt/plugins/vt_joblock.py

if self.lock_file is not None:
    os.unlink(self.lock_file)

Practical Examples

Configuring the Lock Directory

To use a persistent lock directory instead of /tmp, specify the directory via the command line or configuration file:

avocado run --vt-joblock-dir=/var/lock/avocado-vt my_virt_tests.py

Alternatively, set the configuration in avocado.conf:

[plugins.vtjoblock]
dir = /var/lock/avocado-vt

Demonstrating Lock Contention

To observe the locking mechanism in action, start two concurrent jobs using the same lock directory:

# Terminal 1: Start a long-running VT job

avocado run --vt-joblock-dir=/tmp/vt-locks virt_test_suite.yaml &

# Terminal 2: Attempt to start a second job

avocado run --vt-joblock-dir=/tmp/vt-locks another_suite.yaml

The second job will fail immediately with an error message indicating the conflicting PID and lock file path:

FAILURE: File "/tmp/vt-locks/avocado-vt-joblock-<jobid>-1000-abc12345.pid" acquired by PID 12345. (OtherProcessHoldsLockError)

Once the first job completes and its lock file is removed, subsequent jobs can acquire the lock and execute normally.

Summary

• The vt_joblock plugin creates uniquely named lock files in a configurable directory to enforce exclusive access to shared virtualization resources.
• It activates only for jobs containing VirtTest derived tests, preventing unnecessary serialization of non-virtualization workloads.
• The plugin uses pid_exists() from avocado.utils.process to verify whether existing lock files belong to running processes, raising OtherProcessHoldsLockError when conflicts are detected.
• Stale lock cleanup automatically removes files belonging to dead processes, preventing permanent deadlocks after crashes.
• Lock files are removed in post_tests, ensuring resources are released immediately after job completion.

Frequently Asked Questions

What happens if the lock directory is not writable?

If the configured lock directory (default /tmp) is not writable, the plugin will fail during lock file creation with a standard OS permission error. Ensure the directory exists and has appropriate write permissions for the user running the Avocado-VT tests, or specify an alternative directory using --vt-joblock-dir.

Does the vt_joblock plugin affect non-virtualization Avocado tests?

No. The plugin specifically checks for tests derived from the VirtTest class in pre_tests. If the job suite does not contain any VirtTest instances, the _lock() method is never invoked, and the job executes without creating lock files or checking for conflicts.

How does the plugin handle system crashes or unclean shutdowns?

The plugin implements stale lock detection by verifying PID liveness using pid_exists(). If a lock file exists but the PID stored within it no longer corresponds to a running process, the plugin treats the file as stale, logs a warning, and deletes it before attempting to create a new lock. This prevents dead lock files from blocking new jobs indefinitely.

Can multiple users run Avocado-VT jobs simultaneously on the same machine?

Yes, provided they use different lock directories or the same directory with unique user IDs in the lock filename. The lock file pattern includes the OS user ID (%(uid)s), so different users create distinct lock files. However, if multiple users share the same lock directory and the same UID (e.g., running as the same system user), only one job can run at a time. To allow concurrent execution for the same user on different resource sets, configure separate lock directories for each job type.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}