# How to Debug CAPEv2 Service Failures: A Complete Guide to System Log Analysis and Troubleshooting

> Effectively debug CAPEv2 service failures by mastering system log analysis. Explore systemd, application, and analysis logs to pinpoint and resolve issues.

- Repository: [Kevin O'Reilly/capev2](https://github.com/kevoreilly/capev2)
- Tags: how-to-guide
- Published: 2026-03-05

---

**Use `systemctl status` and `journalctl` to inspect systemd-level failures, then examine application logs in `/opt/CAPEv2/log/` and per-analysis logs in `storage/analyses/<id>/` to trace root causes.**

CAPEv2 malware analysis framework runs as a collection of **systemd services** that can fail silently or crash unexpectedly. Understanding how to debug CAPEv2 service failures requires tracing issues through three architectural layers: the **systemd unit definitions**, the Python bootstrap process in [`cuckoo.py`](https://github.com/kevoreilly/capev2/blob/main/cuckoo.py), and the application's logging infrastructure in [`utils/process.py`](https://github.com/kevoreilly/capev2/blob/main/utils/process.py). This guide maps the exact file paths and commands needed to diagnose crashes, permission errors, and processing bottlenecks.

## Systemd-Level Diagnostics

CAPEv2 services like `cape.service`, `cape-processor.service`, and `cape-rooter.service` are controlled by systemd unit files located in the repository's `systemd/` directory. When a service stops unexpectedly, start your investigation at the init system layer.

### Check Service Status and Exit Codes

Run `systemctl` to view the current state, restart count, and recent journal entries:

```bash
systemctl status cape.service

```

This command reveals the **exit code** and whether systemd has triggered the `Restart=always` policy defined in `systemd/cape.service`.

### Inspect the System Journal

To view the complete boot log for the service without pagination:

```bash
journalctl -u cape.service -b --no-pager

```

The `-b` flag limits output to the current boot, which isolates logs after a recent system restart or crash.

### Enable Debug Mode at the Service Level

Force the service to run with verbose logging by setting an environment variable in the systemd unit. Edit `/etc/systemd/system/cape.service` or the repository's `systemd/cape.service` to add:

```ini
Environment=CAPEv2_DEBUG=1
StandardOutput=journal
StandardError=journal

```

Then reload and restart:

```bash
systemctl daemon-reload
systemctl restart cape.service

```

This configuration ensures that [`cuckoo.py`](https://github.com/kevoreilly/capev2/blob/main/cuckoo.py) receives the debug flag, causing `init_logging` to set `log.setLevel(logging.DEBUG)` as implemented in lines 75-78 of [`cuckoo.py`](https://github.com/kevoreilly/capev2/blob/main/cuckoo.py).

## Application-Level Logging Architecture

Once past the systemd layer, CAPEv2 uses Python's standard `logging` module with configurations defined in two critical files.

### Global Logger Configuration

The Django web interface defines its logging behavior in [`web/web/settings.py`](https://github.com/kevoreilly/capev2/blob/main/web/web/settings.py) (lines 79-99). This **LOGGING** dictionary controls whether HTTP 500 errors trigger admin emails. Note that the mail handler disables automatically when `DEBUG=True`, so production troubleshooting requires `DEBUG=False` in this file to receive crash notifications.

### Rotating File Handlers

The [`utils/process.py`](https://github.com/kevoreilly/capev2/blob/main/utils/process.py) file contains the `init_logging` function that builds three handlers: a console handler, an optional syslog handler, and a **rotating file handler**. By default, this writes to `<CUCKOO_ROOT>/log/process.log` with rotation logic around lines 11-15 of the function.

When you run [`cuckoo.py`](https://github.com/kevoreilly/capev2/blob/main/cuckoo.py) with the `-d` flag, the bootstrap script calls `init_logging` with `debug=True`, which sets the log level to `DEBUG` at lines 23-26 of `init_logging`.

### Real-Time Log Monitoring

Monitor the central application log during service startup:

```bash
tail -f /opt/CAPEv2/log/process.log | grep -i "error"

```

## Per-Analysis Log Files

Each analysis task generates isolated logs through `init_per_analysis_logging` in [`utils/process.py`](https://github.com/kevoreilly/capev2/blob/main/utils/process.py) (lines 32-48). These files follow the naming convention `process-<task_id>.log` and reside in one of two locations depending on your configuration:

- **Centralized**: `/opt/CAPEv2/log/process-<task_id>.log`
- **Analysis-specific**: `/opt/CAPEv2/storage/analyses/<task_id>/process.log`

The location is controlled by the `logconf.logger.process_analysis_folder` configuration flag.

To inspect a specific failed analysis:

```bash
TASK_ID=1234
LOG_PATH="/opt/CAPEv2/storage/analyses/${TASK_ID}/process.log"
if [ -f "$LOG_PATH" ]; then
    less "$LOG_PATH"
else
    echo "No per-analysis log - checking central log"
    less "/opt/CAPEv2/log/process.log"
fi

```

## Common Failure Patterns and Diagnostic Signals

| Symptom | Root Cause | Diagnostic Location |
|---------|------------|---------------------|
| **Service repeatedly restarts every 5 minutes** | Systemd `RestartSec=5m` policy triggered by `PermissionError` in `init_logging` (lines 20-21) | `journalctl -u cape.service` for permission denials on `/opt/CAPEv2/log/` |
| **Missing analysis results** | `init_per_analysis_logging` fails to create log files due to missing directories or permissions | [`utils/process.py`](https://github.com/kevoreilly/capev2/blob/main/utils/process.py) lines 54-56 for `PermissionError` exceptions |
| **No HTTP 500 email alerts** | Django `LOGGING` dict disables admin mailer when `DEBUG=True` in [`web/web/settings.py`](https://github.com/kevoreilly/capev2/blob/main/web/web/settings.py) | Verify `DEBUG=False` and mail server configuration |
| **Stuck processing pool** | Unhandled exception in `processing_finished` | Look for `log.exception` tracebacks at line 91 in process logs |
| **Tcpdump permission errors** | `check_tcpdump_permissions` failure during `cuckoo_init` | Early boot logs in [`cuckoo.py`](https://github.com/kevoreilly/capev2/blob/main/cuckoo.py) around line 88 |

## Manual Debugging Commands

When automated service restarts obscure the error, run CAPEv2 manually to capture full debug output:

```bash
cd /opt/CAPEv2
python -d cuckoo.py -m 10

```

The `-d` flag forces debug logging directly to your terminal, bypassing the systemd journal. This is particularly effective for catching import errors or database connection failures during the bootstrap phase in [`cuckoo.py`](https://github.com/kevoreilly/capev2/blob/main/cuckoo.py).

For router-related service issues, examine [`utils/router_manager.py`](https://github.com/kevoreilly/capev2/blob/main/utils/router_manager.py) for additional CLI debug flags that control network routing diagnostics.

## Summary

- **Systemd layer**: Use `systemctl status cape.service` and `journalctl -u cape.service -b` to identify crash loops and exit codes.
- **Bootstrap layer**: Run `python -d cuckoo.py` manually to expose debug output from `cuckoo_init` before the daemon forks.
- **Application layer**: Check rotating logs in `/opt/CAPEv2/log/process.log` configured by [`utils/process.py`](https://github.com/kevoreilly/capev2/blob/main/utils/process.py).
- **Task layer**: Inspect per-analysis logs in `storage/analyses/<task_id>/process.log` generated by `init_per_analysis_logging`.
- **Permissions**: Ensure the `cape` user owns the `log/` directory (chmod 750) and all parent paths are writable.

## Frequently Asked Questions

### Why does my CAPEv2 service keep restarting every 5 minutes?

Systemd's `Restart=always` policy in `systemd/cape.service` triggers this behavior when the Python process exits with an error. Check `journalctl -u cape.service` for `PermissionError` messages from `init_logging` (lines 20-21 of [`utils/process.py`](https://github.com/kevoreilly/capev2/blob/main/utils/process.py)), which typically indicate the `cape` user cannot write to `/opt/CAPEv2/log/` or create the per-analysis log directories.

### How do I enable debug logging for a single analysis without changing the global service configuration?

Run the processing component manually with the debug flag: `python -d utils/process.py <task_id>`. This invokes `init_logging` with `debug=True`, setting `log.setLevel(logging.DEBUG)` and creating verbose output in either the central log or the specific analysis folder depending on your `logconf.logger.process_analysis_folder` setting.

### Where are HTTP 500 errors logged when the web interface crashes?

The Django logging configuration in [`web/web/settings.py`](https://github.com/kevoreilly/capev2/blob/main/web/web/settings.py) (lines 79-99) controls this behavior. By default, HTTP 500 errors trigger admin emails only when `DEBUG=False`. If `DEBUG=True` in your configuration, errors appear only in the console or `web.log` file without email alerts. Check the `LOGGING` dictionary's admin email handler configuration to ensure SMTP settings are correct.

### What causes "Permission denied" errors for tcpdump during CAPEv2 startup?

The `check_tcpdump_permissions` function called during `cuckoo_init` (around line 88 of [`cuckoo.py`](https://github.com/kevoreilly/capev2/blob/main/cuckoo.py)) validates that the cape user can execute tcpdump. These errors appear early in the boot log. Ensure the cape user is a member of the `wireshark` or `pcap` group, or that tcpdump has the appropriate capabilities set via `setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump`.