How to Debug Container Issues Using Boot Logs and System Logs

Use container system logs to inspect daemon-level activity and container logs --boot (or container machine logs --boot) to examine VM initialization failures captured in vminitd.log when debugging the Apple Container stack.

Effective debugging of container failures in the apple/container repository relies on distinguishing between system-level daemon logs and virtual machine boot logs. This guide covers the CLI commands, source implementation details, and correlation strategies needed to diagnose issues ranging from sandbox crashes to early-stage VM initialization failures.

Understanding Log Types in Apple Container

The Container CLI surfaces three distinct output streams, each serving a specific debugging purpose.

System Logs (Daemon-Level)

System logs capture OS-level messages emitted by Container services, including the sandbox daemon, networking plugins, and vminitd. These logs are ideal for diagnosing daemon crashes, entitlement failures, and host-side networking errors. The implementation in Sources/ContainerCommands/System/SystemLogs.swift constructs a filtered log command targeting the com.apple.container subsystem.

Boot Logs (VM Initialization)

Boot logs record the virtual machine's early-boot output before the container's stdio is attached. Written by the vminitd init process to vminitd.log within the machine bundle (as defined in Sources/Services/MachineAPIService/Client/MachineBundle.swift), these logs contain kernel messages and init-script execution details. They are essential for diagnosing init-script failures or early-stage crashes.

Standard I/O Logs (Runtime Output)

After the VM reaches the booted state, stdout and stderr from the container process are available. The CLI accesses these through the same API as boot logs but selects a different file handle index.

Accessing System Logs

The container system logs command interfaces directly with macOS Unified Logging. According to Sources/ContainerCommands/System/SystemLogs.swift, the command builds either a log show or log stream invocation:

var args = ["log"]
args.append(self.follow ? "stream" : "show")
args.append(contentsOf: ["--info", logOptions.debug ? "--debug" : nil].compactMap { $0 })
if !self.follow { args.append(contentsOf: ["--last", last]) }
args.append(contentsOf: ["--predicate", "subsystem = 'com.apple.container'"])

This executes /usr/bin/env log with the specified predicate (lines 71–78 in SystemLogs.swift).

Common usage patterns:

# Show the last 5 minutes of container system logs (default)

container system logs

# Follow live daemon output

container system logs --follow

# Show the last 30 seconds of activity

container system logs --last 30s

System logs are the first place to check when the Container daemon itself misbehaves, such as during sandbox crashes or networking plugin failures.

Inspecting Boot Logs

The boot log resides at vminitd.log inside each machine bundle. Both container machine logs and container logs expose a --boot flag to access this stream.

In Sources/ContainerCommands/Machine/MachineLogs.swift and Sources/ContainerCommands/Container/ContainerLogs.swift, the implementation retrieves two file handles from the API and selects the appropriate index:

let fhs = try await client.logs(id: id)
let fileHandle = boot ? fhs[1] : fhs[0]   // MachineLogs.swift lines 60–62

Here, fhs[1] corresponds to the boot log file handle, while fhs[0] provides standard I/O.

Typical boot log commands:

# View a container's boot log instead of its stdio

container logs --boot mycontainer

# Follow a machine's boot log in real-time

container machine logs --boot --follow mymachine

# Show the last 50 lines of a machine's boot log

container machine logs --boot --lines 50 mymachine

Use boot logs to verify that custom init scripts executed correctly or to identify why a VM failed to reach the booted state.

Correlating Logs for Debugging

A systematic approach to container debugging involves checking these streams in sequence:

1. Start with the boot log to confirm the VM initialized successfully and identify early-stage kernel or init-script errors.

2. Consult system logs if the boot log ends abruptly or shows "failed to start" messages. The system logs reveal why the sandbox daemon terminated, such as missing entitlements or file-system errors.

3. Examine stdio logs (without the --boot flag) once you confirm the VM booted successfully and you need to inspect application-level output.

This correlation strategy prevents manual navigation of bundle directories and accelerates root cause identification.

Practical Debugging Examples

Example 1 – Investigating a Sandbox Crash

When the Container daemon crashes during container creation, check the system logs for the last 10 minutes:

container system logs --last 10m

Look for com.apple.container subsystem messages indicating sandbox initialization failures.

Example 2 – Monitoring Init Script Execution

To watch a custom init script run in real-time during machine startup:

container machine logs --boot --follow mymachine

Press Ctrl+C to stop following when the boot completes.

Example 3 – Diagnosing Failed Container Starts

If container start exits with an error, inspect the boot log immediately:

container logs --boot mycontainer

This reveals vminitd exit codes or kernel panic information not visible in stdio.

Example 4 – Complete Troubleshooting Workflow

# Step 1: Check for VM boot failures

container logs --boot mycontainer

# Step 2: If boot succeeded but issues persist, check daemon health

container system logs --last 5m

# Step 3: Inspect application output

container logs mycontainer

Summary

• System logs (container system logs) filter the com.apple.container subsystem using macOS Unified Logging and are implemented in Sources/ContainerCommands/System/SystemLogs.swift.
• Boot logs (container logs --boot or container machine logs --boot) expose the vminitd.log file from the machine bundle, selected via file handle index fhs[1] in the Machine and Container logs commands.
• Correlation strategy: Boot logs first for initialization issues, system logs for daemon failures, stdio logs for application errors.
• Boot logs physically reside in each machine bundle at vminitd.log as defined in Sources/Services/MachineAPIService/Client/MachineBundle.swift.

Frequently Asked Questions

What is the difference between system logs and boot logs in Apple Container?

System logs capture daemon-level activity from the com.apple.container subsystem using macOS Unified Logging, while boot logs contain the virtual machine's early-boot output written by vminitd to vminitd.log before the container's stdio is attached. System logs show host-side daemon behavior; boot logs show VM initialization sequence.

Where are boot logs physically stored on disk?

Boot logs are stored as vminitd.log inside each machine bundle directory. The path is constructed in Sources/Services/MachineAPIService/Client/MachineBundle.swift using FilePath.Component("vminitd.log") appended to the bundle path.

How do I follow logs in real-time?

Append the --follow flag to either command. For system logs, this switches from log show to log stream. For boot logs, the CLI maintains an open file handle on vminitd.log and streams new content as it arrives from the vminitd process.

Why does my container show no output with container logs but shows data with --boot?

This indicates the container process started but the VM either failed to boot or the application exited before producing stdio. The --boot flag reads from fhs[1] (the boot log file handle), while the default container logs reads from fhs[0] (standard I/O). If the VM crashes during initialization, fhs[0] remains empty while fhs[1] contains the crash details.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}