# How to Troubleshoot Container System Service Startup Failures > Troubleshoot container system service startup failures by checking launchd registration, plist files, service logs, and TOML configuration. Fix your container system now. - Repository: [Apple/container](https://github.com/apple/container) - Tags: tutorial - Published: 2026-07-07 --- **To troubleshoot container system service startup failures, verify launchd registration with `launchctl list`, inspect the generated plist files, check service logs using `container system logs`, and validate the TOML configuration read at launch time.** The `container system start` command in the [apple/container](https://github.com/apple/container) repository launches background services—including the API server, runtime-linux, and network-vmnet—via **launchd**. When these services fail to initialize, the root cause typically lies in launchd registration errors, malformed plist configurations, or missing kernel components. This guide walks through the diagnostic flow implemented in [`Sources/ContainerCommands/System/SystemStart.swift`](https://github.com/apple/container/blob/main/Sources/ContainerCommands/System/SystemStart.swift) to identify and resolve startup issues. ## Verify launchd Service Registration The `SystemStart` command relies on `ServiceManager.register(plistPath:)` to create and register launchd plists. If registration fails, the services never launch. Check whether the services are loaded in launchd: ```bash launchctl list com.apple.container.apiserver launchctl list com.apple.container.runtime launchctl list com.apple.container.network ``` If these commands return "Could not find service," the registration step failed. This usually indicates insufficient privileges or write errors to `$HOME/Library/LaunchAgents`. Run the registration with appropriate permissions or verify the user has access to the LaunchAgents directory. ## Inspect the Generated Plist Files During startup, the CLI generates an `apiserver.plist` (lines 15‑22 in [`SystemStart.swift`](https://github.com/apple/container/blob/main/SystemStart.swift)) that specifies the executable path, arguments, and environment variables. A malformed path or unresolved symlink prevents launchd from executing the binary. View the active plist configuration: ```bash cat $(container system property get apiserver-plist-path) ``` Alternatively, inspect the file directly: ```bash cat $HOME/Library/Containers/com.apple.container.apiserver/apiserver.plist ``` Verify that `ProgramArguments` points to a valid, signed binary. An incorrect path here is the most common cause of silent startup failures. ## Check Service Logs for Errors After registration, services write logs that capture runtime errors. The `container system logs` command reads these files to surface issues like "failed to get a response from apiserver" (referenced in `SystemStart.run` lines 32‑38). Tail logs in real time: ```bash container system logs --follow ``` View recent history only: ```bash container system logs --last 10m ``` Look for errors indicating network conflicts, permission denials, or API server crashes. Messages such as "cannot start network … already started" indicate orphaned processes from previous runs. ## Perform Manual Health Checks The startup sequence includes a health check via `ClientHealthCheck.ping` to verify the API server is responsive. You can trigger this manually to isolate connectivity issues from registration problems. Check system status: ```bash container system status ``` Verify server reachability: ```bash container system version ``` If `container system status` returns "failed to get a response from apiserver" but `launchctl list` shows the service as loaded, the binary likely crashed immediately after launch—check the plist `ProgramArguments` and code signing. ## Validate TOML Configuration Files The loader reads configuration from `$APP_ROOT/config.toml` and `$INSTALL_ROOT/config.toml` in first-match-wins order, as implemented in [`ContainerSystemConfig.swift`](https://github.com/apple/container/blob/main/ContainerSystemConfig.swift). Malformed TOML syntax (missing brackets, type mismatches) aborts startup before services launch. View the merged configuration: ```bash container system property list ``` Inspect the user configuration directly: ```bash cat $HOME/.config/container/config.toml ``` Ensure the file parses as valid TOML. Configuration errors typically manifest as "configuration file parse error" in the logs. ## Verify Kernel Installation and Permissions `SystemStart.run` checks `kernelExists()` (lines 57‑60) to ensure the default Linux kernel is present. A missing or corrupted kernel causes the runtime service to abort immediately after launch. Reinstall the recommended kernel: ```bash container system kernel set --recommended ``` Verify the installation: ```bash container system version ``` Additionally, confirm the API server binary is properly code-signed to satisfy macOS Gatekeeper: ```bash codesign -vvv /usr/local/bin/container-apiserver ``` An unsigned binary will be rejected by launchd, resulting in immediate termination without detailed error messages. ## Run with Debug Output For verbose diagnostics, pass the `--debug` flag to inject `-v` into the launchd arguments (handled in [`SystemStart.swift`](https://github.com/apple/container/blob/main/SystemStart.swift) lines 100‑103). This surfaces low-level registration and runtime details. Start with maximum verbosity: ```bash container system start --debug --enable-kernel-install ``` This output often reveals path resolution issues, environment variable problems, or launchd permission denials that are hidden in normal mode. ## Clean Up Stale Services Orphaned launchd jobs or stale plist files can block new service starts. The `SystemStop` command deregisters services via `ServiceManager`, but manual cleanup is sometimes required. Stop all services: ```bash container system stop ``` Remove stale plists: ```bash rm -rf $HOME/.container/apiserver.plist rm -rf $HOME/.container/network* ``` Then retry: ```bash container system start ``` ## Common Startup Failure Scenarios ### API Server Unresponsive Despite Running Service **Symptom:** `launchctl list` shows the service loaded, but `container system status` returns "failed to get a response from apiserver." **Cause:** The binary path in the plist is incorrect or the binary lacks proper code signing. **Fix:** Verify `ProgramArguments` in the plist (step 2) and ensure the binary is signed with `codesign -vvv`. ### Network Service Conflicts **Symptom:** Logs contain "cannot start network … already started." **Cause:** A previous `container system start` left the network service running or created a zombie process. **Fix:** Run `container system stop`, then remove network-specific plists: `rm -rf $HOME/.container/network*`. ### Configuration Parse Errors **Symptom:** Service refuses to start with "configuration file parse error." **Cause:** Malformed TOML in [`config.toml`](https://github.com/apple/container/blob/main/config.toml) (missing brackets, incorrect data types). **Fix:** Run `container system property list` to view the merged config and correct the syntax in `$HOME/.config/container/config.toml`. ### Missing Kernel After Installation **Symptom:** Health check fails with "invalid state" following a kernel install attempt. **Cause:** The kernel file is missing or corrupted at the expected path. **Fix:** Reinstall using `container system kernel set --recommended` and verify with `container system version`. ## Summary - **Verify launchd registration** with `launchctl list` to ensure `ServiceManager.register(plistPath:)` succeeded. - **Inspect plist files** generated in [`SystemStart.swift`](https://github.com/apple/container/blob/main/SystemStart.swift) to confirm binary paths and arguments are correct. - **Check logs** using `container system logs` to identify runtime crashes or network conflicts. - **Validate configuration** TOML files loaded by [`ContainerSystemConfig.swift`](https://github.com/apple/container/blob/main/ContainerSystemConfig.swift) to prevent parse errors. - **Confirm kernel presence** via `kernelExists()` checks and ensure binaries are code-signed for Gatekeeper compliance. - **Use `--debug`** mode to expose low-level launchd errors and registration details. ## Frequently Asked Questions ### Why does `container system start` fail silently without error messages? When the command fails silently, launchd typically rejected the binary or the plist registration failed. Check `launchctl list` to verify the service exists, then inspect the plist file using `container system property get apiserver-plist-path`. If the service is listed but not responding, verify the binary is code-signed with `codesign -vvv` and that the executable path in the plist resolves correctly. ### How do I reset the container system services to a clean state? Run `container system stop` to deregister services via `ServiceManager`, then manually remove stale plist files from `$HOME/.container/` and `$HOME/Library/LaunchAgents/`. Delete the apiserver plist specifically with `rm -rf $HOME/.container/apiserver.plist`. After cleanup, re-run `container system start` to generate fresh configurations. ### Where are the configuration files loaded from during startup? The system reads [`config.toml`](https://github.com/apple/container/blob/main/config.toml) from `$APP_ROOT/config.toml` and `$INSTALL_ROOT/config.toml` in first-match-wins order, as implemented in [`ContainerSystemConfig.swift`](https://github.com/apple/container/blob/main/ContainerSystemConfig.swift). User-specific settings can be placed in `$HOME/.config/container/config.toml`. Use `container system property list` to view the merged configuration currently in use. ### What causes the "failed to get a response from apiserver" error during startup? This error occurs when the API server process crashes immediately after launchd starts it. Common causes include an invalid `ProgramArguments` path in the plist, an unsigned binary blocked by Gatekeeper, or a missing kernel file detected by `kernelExists()`. Verify the plist content, check code signing, and ensure the kernel is installed using `container system kernel set --recommended`.