How Apple Container Integrates with the Virtualization Framework

Apple Container leverages the Virtualization framework's VZVirtualMachineManager to create Linux VMs that host containers, while using NonisolatedInterfaceStrategy with vmnet to bridge custom network interfaces into the virtualized environment.

The Apple Container project runs each container inside a lightweight Linux virtual machine on macOS. According to the source code in the apple/container repository, the Virtualization framework integration spans VM lifecycle management, network bridging, and filesystem mounting to provide isolated container environments.

VM Creation via VZVirtualMachineManager

When the runtime service bootstraps a container, it instantiates a VZVirtualMachineManager from the Virtualization framework to handle the VM lifecycle. In Sources/Services/RuntimeLinux/Server/RuntimeService.swift, the service creates the manager with the container's kernel and root filesystem:

let vmm = VZVirtualMachineManager(
    kernel: kernel,
    initialFilesystem: bundle.initialFilesystem.asMount,
    rosetta: config.rosetta,
    logger: self.log
)

Source: RuntimeService.swift – lines 165‑167

The VZVirtualMachineManager accepts several key parameters:

• kernel: The Linux kernel binary provided by the container bundle
• initialFilesystem: The root filesystem mount (often virtiofs or tmpfs) that becomes the VM's /
• rosetta: Enables Rosetta translation for running x86_64 code on Apple Silicon
• logger: Logging infrastructure for debugging

After creating the manager, the service wraps it in a LinuxContainer instance:

let container = try LinuxContainer(
    id,
    rootfs: bundle.containerRootfs.asMount,
    vmm: vmm,
    logger: self.log
) { czConfig in
    // configure container (CPU, memory, mounts, etc.)
}

Source: RuntimeService.swift – lines 53‑56

This architecture delegates all VM lifecycle operations—start, stop, snapshot, and state management—to the Virtualization framework through the VZVirtualMachine objects managed by LinuxContainer.

Network Integration with NonisolatedInterfaceStrategy

For containers requesting non-isolated networks (shared with the host), the runtime uses NonisolatedInterfaceStrategy to bridge the Virtualization framework with vmnet. This strategy deserializes network references and creates NATNetworkInterface instances that the VM attaches to.

In Sources/Services/RuntimeLinux/Server/NonisolatedInterfaceStrategy.swift, the toInterface method handles the integration:

public func toInterface(
    attachment: Attachment,
    interfaceIndex: Int,
    additionalData: XPCMessage?
) throws -> Interface {
    // Deserialize the vmnet network reference received from the network plugin
    var status: vmnet_return_t = .VMNET_SUCCESS
    guard let networkRef = vmnet_network_create_with_serialization(
        additionalData.underlying, &status) else {
        throw ContainerizationError(...)
    }
    // Build a NATNetworkInterface that the VM will see
    return NATNetworkInterface(
        ipv4Address: attachment.ipv4Address,
        ipv4Gateway: ipv4Gateway,
        reference: networkRef,
        macAddress: attachment.macAddress,
        mtu: attachment.mtu ?? 1280
    )
}

Source: NonisolatedInterfaceStrategy.swift – lines 23‑55

The integration follows this sequence:

1. Import Virtualization: The framework provides APIs for VZVirtualMachine networking
2. Deserialize: vmnet_network_create_with_serialization reconstructs the network reference from the plugin's XPC message
3. Expose: The returned NATNetworkInterface attaches to the VM as a virtual NIC backed by the host's vmnet network

This allows the container to access the host-managed network while maintaining virtualization boundaries.

Filesystem Caching for Virtualization Stability

When using virtiofs for root filesystem or data shares, Apple Container defaults the cache mode to .on to avoid a known Linux filesystem bug that manifests under virtualization. This configuration appears in Sources/ContainerResource/Container/Filesystem.swift:

// Defaulting to CachedMode = .on (i.e., cached mode) to fix Linux FS issue when using Virtualization

Source: Filesystem.swift – lines 88‑90

This workaround references issue #614 and ensures stable filesystem operations when the Virtualization framework handles the underlying storage devices.

End-to-End Container Initialization Flow

The complete integration between container and Virtualization framework follows this orchestrated sequence:

1. Client Request: An XPC client asks the runtime service to bootstrap a container
2. VM Construction: RuntimeService.bootstrap creates the VZVirtualMachineManager with the bundle's kernel and initial filesystem
3. Container Wrapping: The service instantiates LinuxContainer with the VM manager and root filesystem
4. Network Setup: For non-isolated networks, NonisolatedInterfaceStrategy creates a NATNetworkInterface backed by a deserialized vmnet reference
5. VM Startup: The Virtualization framework starts the VM, mounts the root filesystem, configures the network interface, and runs the init process
6. Lifecycle Management: Throughout the container's lifetime, the service uses Virtualization APIs (e.g., container.stop(), container.dialVsock()) to control the VM

Summary

• VZVirtualMachineManager creates and manages the Linux VM that hosts each container, handling kernel loading and filesystem mounting according to the apple/container source code
• NonisolatedInterfaceStrategy bridges the Virtualization framework with vmnet to expose host-managed networks to containers through NATNetworkInterface instances
• Filesystem caching defaults to .on for virtiofs mounts to prevent Linux filesystem bugs under virtualization
• The runtime service orchestrates VM lifecycle, networking, and container configuration through the Virtualization framework APIs implemented in RuntimeService.swift

Frequently Asked Questions

How does the Virtualization framework improve container isolation?

The Virtualization framework provides hardware-level isolation by running each container inside a dedicated Linux VM. Unlike traditional Linux containers that share the host kernel, Apple Container uses VZVirtualMachineManager to create separate virtual machines with their own kernels and namespaces, ensuring stronger security boundaries while maintaining lightweight performance characteristics.

What network types does Apple Container support through the Virtualization framework?

Apple Container supports isolated and non-isolated network modes. For non-isolated networks, the NonisolatedInterfaceStrategy uses the Virtualization framework alongside vmnet to create NATNetworkInterface instances. These interfaces allow containers to share the host's network stack while maintaining virtualization boundaries, configured through the container-network-vmnet plugin.

Why is Rosetta support included in the VM configuration?

The rosetta parameter in VZVirtualMachineManager enables x86_64 binary translation on Apple Silicon Macs. When enabled, the Virtualization framework provides Rosetta support within the Linux VM, allowing containers built for Intel architecture to run natively on ARM-based Apple hardware without requiring separate container images.

Where does the Linux kernel come from for these containers?

The Linux kernel binary is provided by the container bundle and passed to VZVirtualMachineManager during initialization. The bundle includes the specific kernel version required for the container environment, which the Virtualization framework loads into the VM alongside the initial root filesystem mount specified in the configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}