# How to Debug Issues Within Containers Running on Apple's Runtime

> Debug containers on Apple's runtime with `--debug` flag. Inspect logs via `container logs --boot`, `container logs`, and `container system logs` to quickly isolate and resolve issues.

- Repository: [Apple/container](https://github.com/apple/container)
- Tags: how-to-guide
- Published: 2026-07-01

---

**Use the `--debug` flag to enable verbose logging across the VM boot, container process, and Runtime Service layers, then inspect logs via `container logs --boot`, `container logs`, and `container system logs` to isolate failures.**

Apple's `container` tool runs each OCI image inside a lightweight virtual machine managed by the **Runtime Service** ([`RuntimeService.swift`](https://github.com/apple/container/blob/main/RuntimeService.swift)). When containers fail to start or behave unexpectedly, you need visibility into three distinct layers: the VM boot sequence, the container process itself, and the XPC control plane. This guide shows how to debug issues within containers running on Apple's runtime using the native diagnostic tools built into the CLI.

## Understanding the Debug Architecture

The container runtime separates concerns across three diagnostic viewpoints. Each layer produces distinct logs that help isolate different classes of failures.

### The Three Diagnostic Layers

Apple's runtime architecture provides three natural "viewpoints" for troubleshooting:

- **VM Boot**: Captures kernel boot messages, early init (`vminitd`) output, and hardware enumeration. Access via `container logs --boot <container-id>`.
- **Container Process**: Shows application stdout/stderr, exit status, and `exec`-attached streams. Access via `container logs <container-id>`.
- **Runtime Service**: Exposes XPC-level tracing, network attachment steps, and internal state changes. Enabled automatically when `--debug` is passed; visible in system logs.

This separation means kernel-level failures surface in boot logs, application errors appear in container logs, and control-plane issues manifest in Runtime Service traces.

### How Debug Logging Works

In [`Sources/Services/RuntimeLinux/Server/RuntimeService.swift`](https://github.com/apple/container/blob/main/Sources/Services/RuntimeLinux/Server/RuntimeService.swift), the service initializes a logger at line 49 that defaults to **info** level during normal operation. When you pass `--debug` to any `container` sub-command, the CLI sets `CONTAINER_DEBUG=1` internally and upgrades the logging level to **debug**.

This enables entry-point tracing (`enter`/`exit`) for every XPC handler. At lines 122-124, each public method such as `createEndpoint`, `bootstrap`, and `startProcess` logs entry and exit points with richer metadata. Network-attachment logic around lines 200-205 also emits detailed diagnostics, making it possible to trace failures in interface strategy or DNS configuration.

## Enabling Debug Mode

Debug output is controlled via a single CLI flag that propagates through the entire stack.

### Using the `--debug` Flag

Add `--debug` to any command to activate verbose tracing:

```bash
container run --debug --name my-web-server registry.example.com/fido/web-test:latest

```

The flag upgrades logging across all three layers. In the container log output, you'll see a "debug" prefix added to messages. The Runtime Service writes detailed XPC call traces to the host system log, while the VM boot log captures verbose kernel initialization messages.

## Collecting and Analyzing Logs

Once debug mode is enabled, you collect logs from the appropriate viewpoint based on where the failure occurs.

### Inspecting VM Boot Logs

Boot logs reveal why a container fails to start, such as missing kernel arguments or hardware enumeration errors. These low-level VM/kernel messages include output from the `vminitd` init process.

```bash
container logs --boot my-web-server | grep -i debug

```

According to the documentation in [`docs/how-to.md`](https://github.com/apple/container/blob/main/docs/how-to.md), typical boot-log entries include kernel initialization and Runtime Service debug messages (lines 84-90).

### Viewing Container Process Logs

After the VM boots successfully, inspect the application stdout/stderr stream. This captures output from the init process and any `exec`-style subprocesses.

```bash
container logs my-web-server

```

When running with `--debug`, these logs include additional prefixes that help correlate container output with Runtime Service events.

### Reading Runtime Service Traces

The Runtime Service logs entry/exit for each XPC call, helping pinpoint where failures occur in network attachment or process startup.

```bash
container system logs | grep -i debug | tail -20

```

These messages flow from [`RuntimeService.swift`](https://github.com/apple/container/blob/main/RuntimeService.swift) and include detailed traces of methods like `createEndpoint` and `startProcess`. If you see errors around network configuration, check the logic at lines 200-205 where interface strategies are validated.

## Advanced Debugging Techniques

When logs alone don't reveal the root cause, you can attach native debuggers directly to the runtime processes.

### Attaching a Native Debugger

Locate the PID of the `container-runtime-linux` helper from the system log:

```bash
PID=$(container system logs | grep "container-runtime-linux" | awk '{print $2}')
lldb -p $PID

```

Alternatively, use Xcode's "Attach to Process" UI to step through the Swift code in [`RuntimeService.swift`](https://github.com/apple/container/blob/main/RuntimeService.swift) while the container is running. This is useful for debugging complex XPC communication failures or network attachment issues that don't produce clear log errors.

### Correlating with Source Code

Debug messages reference specific locations in [`Sources/Services/RuntimeLinux/Server/RuntimeService.swift`](https://github.com/apple/container/blob/main/Sources/Services/RuntimeLinux/Server/RuntimeService.swift):

- Entry/exit tracing for public methods: lines 122-124
- Logger initialization and level configuration: line 49
- Network attachment logic: lines 200-205

Cross-reference log timestamps with these code locations to understand exactly which initialization step failed.

## Summary

To debug issues within containers running on Apple's runtime:

- **Enable verbose logging** with the `--debug` flag to activate debug-level output across VM boot, container process, and Runtime Service layers.
- **Check boot logs first** using `container logs --boot <id>` to identify kernel or initialization failures referenced in [`docs/how-to.md`](https://github.com/apple/container/blob/main/docs/how-to.md).
- **Inspect application output** via `container logs <id>` for stdout/stderr errors and exit statuses.
- **Trace XPC calls** in `container system logs` to diagnose control-plane and network attachment issues in [`RuntimeService.swift`](https://github.com/apple/container/blob/main/RuntimeService.swift).
- **Attach native debuggers** like `lldb` to the `container-runtime-linux` process for interactive debugging of the Swift source.

## Frequently Asked Questions

### How do I enable debug mode for Apple container commands?

Add the `--debug` flag to any `container` sub-command (such as `run`, `start`, or `exec`). This sets `CONTAINER_DEBUG=1` internally and upgrades the Runtime Service logger from info to debug level, enabling entry/exit tracing for all XPC handlers in [`RuntimeService.swift`](https://github.com/apple/container/blob/main/RuntimeService.swift).

### Where are the Runtime Service logs stored?

The Runtime Service writes debug logs to the host-side system log, accessible via `container system logs`. These logs include XPC-level tracing, network attachment steps, and internal state changes from [`RuntimeService.swift`](https://github.com/apple/container/blob/main/RuntimeService.swift), particularly around lines 122-124 where entry/exit points are logged.

### What is the difference between boot logs and container logs?

Boot logs (`container logs --boot`) contain VM-level output including kernel messages and `vminitd` initialization, useful for diagnosing startup failures. Container logs (`container logs`) capture the stdout/stderr of the container process and any attached exec sessions, useful for application-level debugging.

### Can I debug the Runtime Service while it's running?

Yes. Locate the `container-runtime-linux` process ID in the system logs, then attach `lldb` or Xcode's debugger to step through the Swift source code. This allows interactive debugging of network attachment logic (around lines 200-205) and XPC method implementations in [`RuntimeService.swift`](https://github.com/apple/container/blob/main/RuntimeService.swift).