How to Debug CAPEv2 Service Failures: A Complete Guide to System Log Analysis and Troubleshooting

Use systemctl status and journalctl to inspect systemd-level failures, then examine application logs in /opt/CAPEv2/log/ and per-analysis logs in storage/analyses/<id>/ to trace root causes.

CAPEv2 malware analysis framework runs as a collection of systemd services that can fail silently or crash unexpectedly. Understanding how to debug CAPEv2 service failures requires tracing issues through three architectural layers: the systemd unit definitions, the Python bootstrap process in cuckoo.py, and the application's logging infrastructure in utils/process.py. This guide maps the exact file paths and commands needed to diagnose crashes, permission errors, and processing bottlenecks.

Systemd-Level Diagnostics

CAPEv2 services like cape.service, cape-processor.service, and cape-rooter.service are controlled by systemd unit files located in the repository's systemd/ directory. When a service stops unexpectedly, start your investigation at the init system layer.

Check Service Status and Exit Codes

Run systemctl to view the current state, restart count, and recent journal entries:

systemctl status cape.service

This command reveals the exit code and whether systemd has triggered the Restart=always policy defined in systemd/cape.service.

Inspect the System Journal

To view the complete boot log for the service without pagination:

journalctl -u cape.service -b --no-pager

The -b flag limits output to the current boot, which isolates logs after a recent system restart or crash.

Enable Debug Mode at the Service Level

Force the service to run with verbose logging by setting an environment variable in the systemd unit. Edit /etc/systemd/system/cape.service or the repository's systemd/cape.service to add:

Environment=CAPEv2_DEBUG=1
StandardOutput=journal
StandardError=journal

Then reload and restart:

systemctl daemon-reload
systemctl restart cape.service

This configuration ensures that cuckoo.py receives the debug flag, causing init_logging to set log.setLevel(logging.DEBUG) as implemented in lines 75-78 of cuckoo.py.

Application-Level Logging Architecture

Once past the systemd layer, CAPEv2 uses Python's standard logging module with configurations defined in two critical files.

Global Logger Configuration

The Django web interface defines its logging behavior in web/web/settings.py (lines 79-99). This LOGGING dictionary controls whether HTTP 500 errors trigger admin emails. Note that the mail handler disables automatically when DEBUG=True, so production troubleshooting requires DEBUG=False in this file to receive crash notifications.

Rotating File Handlers

The utils/process.py file contains the init_logging function that builds three handlers: a console handler, an optional syslog handler, and a rotating file handler. By default, this writes to <CUCKOO_ROOT>/log/process.log with rotation logic around lines 11-15 of the function.

When you run cuckoo.py with the -d flag, the bootstrap script calls init_logging with debug=True, which sets the log level to DEBUG at lines 23-26 of init_logging.

Real-Time Log Monitoring

Monitor the central application log during service startup:

tail -f /opt/CAPEv2/log/process.log | grep -i "error"

Per-Analysis Log Files

Each analysis task generates isolated logs through init_per_analysis_logging in utils/process.py (lines 32-48). These files follow the naming convention process-<task_id>.log and reside in one of two locations depending on your configuration:

• Centralized: /opt/CAPEv2/log/process-<task_id>.log
• Analysis-specific: /opt/CAPEv2/storage/analyses/<task_id>/process.log

The location is controlled by the logconf.logger.process_analysis_folder configuration flag.

To inspect a specific failed analysis:

TASK_ID=1234
LOG_PATH="/opt/CAPEv2/storage/analyses/${TASK_ID}/process.log"
if [ -f "$LOG_PATH" ]; then
    less "$LOG_PATH"
else
    echo "No per-analysis log - checking central log"
    less "/opt/CAPEv2/log/process.log"
fi

Common Failure Patterns and Diagnostic Signals

Symptom	Root Cause	Diagnostic Location
Service repeatedly restarts every 5 minutes	Systemd RestartSec=5m policy triggered by PermissionError in init_logging (lines 20-21)	journalctl -u cape.service for permission denials on /opt/CAPEv2/log/
Missing analysis results	init_per_analysis_logging fails to create log files due to missing directories or permissions	utils/process.py lines 54-56 for PermissionError exceptions
No HTTP 500 email alerts	Django LOGGING dict disables admin mailer when DEBUG=True in web/web/settings.py	Verify DEBUG=False and mail server configuration
Stuck processing pool	Unhandled exception in processing_finished	Look for log.exception tracebacks at line 91 in process logs
Tcpdump permission errors	check_tcpdump_permissions failure during cuckoo_init	Early boot logs in cuckoo.py around line 88

Manual Debugging Commands

When automated service restarts obscure the error, run CAPEv2 manually to capture full debug output:

cd /opt/CAPEv2
python -d cuckoo.py -m 10

The -d flag forces debug logging directly to your terminal, bypassing the systemd journal. This is particularly effective for catching import errors or database connection failures during the bootstrap phase in cuckoo.py.

For router-related service issues, examine utils/router_manager.py for additional CLI debug flags that control network routing diagnostics.

Summary

• Systemd layer: Use systemctl status cape.service and journalctl -u cape.service -b to identify crash loops and exit codes.
• Bootstrap layer: Run python -d cuckoo.py manually to expose debug output from cuckoo_init before the daemon forks.
• Application layer: Check rotating logs in /opt/CAPEv2/log/process.log configured by utils/process.py.
• Task layer: Inspect per-analysis logs in storage/analyses/<task_id>/process.log generated by init_per_analysis_logging.
• Permissions: Ensure the cape user owns the log/ directory (chmod 750) and all parent paths are writable.

Frequently Asked Questions

Why does my CAPEv2 service keep restarting every 5 minutes?

Systemd's Restart=always policy in systemd/cape.service triggers this behavior when the Python process exits with an error. Check journalctl -u cape.service for PermissionError messages from init_logging (lines 20-21 of utils/process.py), which typically indicate the cape user cannot write to /opt/CAPEv2/log/ or create the per-analysis log directories.

How do I enable debug logging for a single analysis without changing the global service configuration?

Run the processing component manually with the debug flag: python -d utils/process.py <task_id>. This invokes init_logging with debug=True, setting log.setLevel(logging.DEBUG) and creating verbose output in either the central log or the specific analysis folder depending on your logconf.logger.process_analysis_folder setting.

Where are HTTP 500 errors logged when the web interface crashes?

The Django logging configuration in web/web/settings.py (lines 79-99) controls this behavior. By default, HTTP 500 errors trigger admin emails only when DEBUG=False. If DEBUG=True in your configuration, errors appear only in the console or web.log file without email alerts. Check the LOGGING dictionary's admin email handler configuration to ensure SMTP settings are correct.

What causes "Permission denied" errors for tcpdump during CAPEv2 startup?

The check_tcpdump_permissions function called during cuckoo_init (around line 88 of cuckoo.py) validates that the cape user can execute tcpdump. These errors appear early in the boot log. Ensure the cape user is a member of the wireshark or pcap group, or that tcpdump has the appropriate capabilities set via setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}