How to Debug Issues Within Containers Running on Apple's Runtime

Use the --debug flag to enable verbose logging across the VM boot, container process, and Runtime Service layers, then inspect logs via container logs --boot, container logs, and container system logs to isolate failures.

Apple's container tool runs each OCI image inside a lightweight virtual machine managed by the Runtime Service (RuntimeService.swift). When containers fail to start or behave unexpectedly, you need visibility into three distinct layers: the VM boot sequence, the container process itself, and the XPC control plane. This guide shows how to debug issues within containers running on Apple's runtime using the native diagnostic tools built into the CLI.

Understanding the Debug Architecture

The container runtime separates concerns across three diagnostic viewpoints. Each layer produces distinct logs that help isolate different classes of failures.

The Three Diagnostic Layers

Apple's runtime architecture provides three natural "viewpoints" for troubleshooting:

• VM Boot: Captures kernel boot messages, early init (vminitd) output, and hardware enumeration. Access via container logs --boot <container-id>.
• Container Process: Shows application stdout/stderr, exit status, and exec-attached streams. Access via container logs <container-id>.
• Runtime Service: Exposes XPC-level tracing, network attachment steps, and internal state changes. Enabled automatically when --debug is passed; visible in system logs.

This separation means kernel-level failures surface in boot logs, application errors appear in container logs, and control-plane issues manifest in Runtime Service traces.

How Debug Logging Works

In Sources/Services/RuntimeLinux/Server/RuntimeService.swift, the service initializes a logger at line 49 that defaults to info level during normal operation. When you pass --debug to any container sub-command, the CLI sets CONTAINER_DEBUG=1 internally and upgrades the logging level to debug.

This enables entry-point tracing (enter/exit) for every XPC handler. At lines 122-124, each public method such as createEndpoint, bootstrap, and startProcess logs entry and exit points with richer metadata. Network-attachment logic around lines 200-205 also emits detailed diagnostics, making it possible to trace failures in interface strategy or DNS configuration.

Enabling Debug Mode

Debug output is controlled via a single CLI flag that propagates through the entire stack.

Using the --debug Flag

Add --debug to any command to activate verbose tracing:

container run --debug --name my-web-server registry.example.com/fido/web-test:latest

The flag upgrades logging across all three layers. In the container log output, you'll see a "debug" prefix added to messages. The Runtime Service writes detailed XPC call traces to the host system log, while the VM boot log captures verbose kernel initialization messages.

Collecting and Analyzing Logs

Once debug mode is enabled, you collect logs from the appropriate viewpoint based on where the failure occurs.

Inspecting VM Boot Logs

Boot logs reveal why a container fails to start, such as missing kernel arguments or hardware enumeration errors. These low-level VM/kernel messages include output from the vminitd init process.

container logs --boot my-web-server | grep -i debug

According to the documentation in docs/how-to.md, typical boot-log entries include kernel initialization and Runtime Service debug messages (lines 84-90).

Viewing Container Process Logs

After the VM boots successfully, inspect the application stdout/stderr stream. This captures output from the init process and any exec-style subprocesses.

container logs my-web-server

When running with --debug, these logs include additional prefixes that help correlate container output with Runtime Service events.

Reading Runtime Service Traces

The Runtime Service logs entry/exit for each XPC call, helping pinpoint where failures occur in network attachment or process startup.

container system logs | grep -i debug | tail -20

These messages flow from RuntimeService.swift and include detailed traces of methods like createEndpoint and startProcess. If you see errors around network configuration, check the logic at lines 200-205 where interface strategies are validated.

Advanced Debugging Techniques

When logs alone don't reveal the root cause, you can attach native debuggers directly to the runtime processes.

Attaching a Native Debugger

Locate the PID of the container-runtime-linux helper from the system log:

PID=$(container system logs | grep "container-runtime-linux" | awk '{print $2}')
lldb -p $PID

Alternatively, use Xcode's "Attach to Process" UI to step through the Swift code in RuntimeService.swift while the container is running. This is useful for debugging complex XPC communication failures or network attachment issues that don't produce clear log errors.

Correlating with Source Code

Debug messages reference specific locations in Sources/Services/RuntimeLinux/Server/RuntimeService.swift:

• Entry/exit tracing for public methods: lines 122-124
• Logger initialization and level configuration: line 49
• Network attachment logic: lines 200-205

Cross-reference log timestamps with these code locations to understand exactly which initialization step failed.

Summary

To debug issues within containers running on Apple's runtime:

• Enable verbose logging with the --debug flag to activate debug-level output across VM boot, container process, and Runtime Service layers.
• Check boot logs first using container logs --boot <id> to identify kernel or initialization failures referenced in docs/how-to.md.
• Inspect application output via container logs <id> for stdout/stderr errors and exit statuses.
• Trace XPC calls in container system logs to diagnose control-plane and network attachment issues in RuntimeService.swift.
• Attach native debuggers like lldb to the container-runtime-linux process for interactive debugging of the Swift source.

Frequently Asked Questions

How do I enable debug mode for Apple container commands?

Add the --debug flag to any container sub-command (such as run, start, or exec). This sets CONTAINER_DEBUG=1 internally and upgrades the Runtime Service logger from info to debug level, enabling entry/exit tracing for all XPC handlers in RuntimeService.swift.

Where are the Runtime Service logs stored?

The Runtime Service writes debug logs to the host-side system log, accessible via container system logs. These logs include XPC-level tracing, network attachment steps, and internal state changes from RuntimeService.swift, particularly around lines 122-124 where entry/exit points are logged.

What is the difference between boot logs and container logs?

Boot logs (container logs --boot) contain VM-level output including kernel messages and vminitd initialization, useful for diagnosing startup failures. Container logs (container logs) capture the stdout/stderr of the container process and any attached exec sessions, useful for application-level debugging.

Can I debug the Runtime Service while it's running?

Yes. Locate the container-runtime-linux process ID in the system logs, then attach lldb or Xcode's debugger to step through the Swift source code. This allows interactive debugging of network attachment logic (around lines 200-205) and XPC method implementations in RuntimeService.swift.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}