How to Troubleshoot Container System Service Startup Failures

To troubleshoot container system service startup failures, verify launchd registration with launchctl list, inspect the generated plist files, check service logs using container system logs, and validate the TOML configuration read at launch time.

The container system start command in the apple/container repository launches background services—including the API server, runtime-linux, and network-vmnet—via launchd. When these services fail to initialize, the root cause typically lies in launchd registration errors, malformed plist configurations, or missing kernel components. This guide walks through the diagnostic flow implemented in Sources/ContainerCommands/System/SystemStart.swift to identify and resolve startup issues.

Verify launchd Service Registration

The SystemStart command relies on ServiceManager.register(plistPath:) to create and register launchd plists. If registration fails, the services never launch.

Check whether the services are loaded in launchd:

launchctl list com.apple.container.apiserver
launchctl list com.apple.container.runtime
launchctl list com.apple.container.network

If these commands return "Could not find service," the registration step failed. This usually indicates insufficient privileges or write errors to $HOME/Library/LaunchAgents. Run the registration with appropriate permissions or verify the user has access to the LaunchAgents directory.

Inspect the Generated Plist Files

During startup, the CLI generates an apiserver.plist (lines 15‑22 in SystemStart.swift) that specifies the executable path, arguments, and environment variables. A malformed path or unresolved symlink prevents launchd from executing the binary.

View the active plist configuration:

cat $(container system property get apiserver-plist-path)

Alternatively, inspect the file directly:

cat $HOME/Library/Containers/com.apple.container.apiserver/apiserver.plist

Verify that ProgramArguments points to a valid, signed binary. An incorrect path here is the most common cause of silent startup failures.

Check Service Logs for Errors

After registration, services write logs that capture runtime errors. The container system logs command reads these files to surface issues like "failed to get a response from apiserver" (referenced in SystemStart.run lines 32‑38).

Tail logs in real time:

container system logs --follow

View recent history only:

container system logs --last 10m

Look for errors indicating network conflicts, permission denials, or API server crashes. Messages such as "cannot start network … already started" indicate orphaned processes from previous runs.

Perform Manual Health Checks

The startup sequence includes a health check via ClientHealthCheck.ping to verify the API server is responsive. You can trigger this manually to isolate connectivity issues from registration problems.

Check system status:

container system status

Verify server reachability:

container system version

If container system status returns "failed to get a response from apiserver" but launchctl list shows the service as loaded, the binary likely crashed immediately after launch—check the plist ProgramArguments and code signing.

Validate TOML Configuration Files

The loader reads configuration from $APP_ROOT/config.toml and $INSTALL_ROOT/config.toml in first-match-wins order, as implemented in ContainerSystemConfig.swift. Malformed TOML syntax (missing brackets, type mismatches) aborts startup before services launch.

View the merged configuration:

container system property list

Inspect the user configuration directly:

cat $HOME/.config/container/config.toml

Ensure the file parses as valid TOML. Configuration errors typically manifest as "configuration file parse error" in the logs.

Verify Kernel Installation and Permissions

SystemStart.run checks kernelExists() (lines 57‑60) to ensure the default Linux kernel is present. A missing or corrupted kernel causes the runtime service to abort immediately after launch.

Reinstall the recommended kernel:

container system kernel set --recommended

Verify the installation:

container system version

Additionally, confirm the API server binary is properly code-signed to satisfy macOS Gatekeeper:

codesign -vvv /usr/local/bin/container-apiserver

An unsigned binary will be rejected by launchd, resulting in immediate termination without detailed error messages.

Run with Debug Output

For verbose diagnostics, pass the --debug flag to inject -v into the launchd arguments (handled in SystemStart.swift lines 100‑103). This surfaces low-level registration and runtime details.

Start with maximum verbosity:

container system start --debug --enable-kernel-install

This output often reveals path resolution issues, environment variable problems, or launchd permission denials that are hidden in normal mode.

Clean Up Stale Services

Orphaned launchd jobs or stale plist files can block new service starts. The SystemStop command deregisters services via ServiceManager, but manual cleanup is sometimes required.

Stop all services:

container system stop

Remove stale plists:

rm -rf $HOME/.container/apiserver.plist
rm -rf $HOME/.container/network*

Then retry:

container system start

Common Startup Failure Scenarios

API Server Unresponsive Despite Running Service

Symptom: launchctl list shows the service loaded, but container system status returns "failed to get a response from apiserver."

Cause: The binary path in the plist is incorrect or the binary lacks proper code signing.

Fix: Verify ProgramArguments in the plist (step 2) and ensure the binary is signed with codesign -vvv.

Network Service Conflicts

Symptom: Logs contain "cannot start network … already started."

Cause: A previous container system start left the network service running or created a zombie process.

Fix: Run container system stop, then remove network-specific plists: rm -rf $HOME/.container/network*.

Configuration Parse Errors

Symptom: Service refuses to start with "configuration file parse error."

Cause: Malformed TOML in config.toml (missing brackets, incorrect data types).

Fix: Run container system property list to view the merged config and correct the syntax in $HOME/.config/container/config.toml.

Missing Kernel After Installation

Symptom: Health check fails with "invalid state" following a kernel install attempt.

Cause: The kernel file is missing or corrupted at the expected path.

Fix: Reinstall using container system kernel set --recommended and verify with container system version.

Summary

• Verify launchd registration with launchctl list to ensure ServiceManager.register(plistPath:) succeeded.
• Inspect plist files generated in SystemStart.swift to confirm binary paths and arguments are correct.
• Check logs using container system logs to identify runtime crashes or network conflicts.
• Validate configuration TOML files loaded by ContainerSystemConfig.swift to prevent parse errors.
• Confirm kernel presence via kernelExists() checks and ensure binaries are code-signed for Gatekeeper compliance.
• Use --debug mode to expose low-level launchd errors and registration details.

Frequently Asked Questions

Why does container system start fail silently without error messages?

When the command fails silently, launchd typically rejected the binary or the plist registration failed. Check launchctl list to verify the service exists, then inspect the plist file using container system property get apiserver-plist-path. If the service is listed but not responding, verify the binary is code-signed with codesign -vvv and that the executable path in the plist resolves correctly.

How do I reset the container system services to a clean state?

Run container system stop to deregister services via ServiceManager, then manually remove stale plist files from $HOME/.container/ and $HOME/Library/LaunchAgents/. Delete the apiserver plist specifically with rm -rf $HOME/.container/apiserver.plist. After cleanup, re-run container system start to generate fresh configurations.

Where are the configuration files loaded from during startup?

The system reads config.toml from $APP_ROOT/config.toml and $INSTALL_ROOT/config.toml in first-match-wins order, as implemented in ContainerSystemConfig.swift. User-specific settings can be placed in $HOME/.config/container/config.toml. Use container system property list to view the merged configuration currently in use.

What causes the "failed to get a response from apiserver" error during startup?

This error occurs when the API server process crashes immediately after launchd starts it. Common causes include an invalid ProgramArguments path in the plist, an unsigned binary blocked by Gatekeeper, or a missing kernel file detected by kernelExists(). Verify the plist content, check code signing, and ensure the kernel is installed using container system kernel set --recommended.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}