# How to Configure the Maximum Concurrent Threads for SpiderFoot Module Execution

> Learn how to configure maximum concurrent threads for SpiderFoot module execution using CLI flags, config files, or the default config dictionary for optimized performance.

- Repository: [Steve Micallef/spiderfoot](https://github.com/smicallef/spiderfoot)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Set the `_maxthreads` parameter via CLI flag (`-max-threads`), configuration file, or the default config dictionary in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) to control how many SpiderFoot modules run simultaneously.**

SpiderFoot executes its scanning modules in parallel using a shared thread pool, and the maximum number of concurrent threads directly impacts scan performance and system resource consumption. This guide explains three methods to configure this setting based on the `smicallef/spiderfoot` source code implementation.

## Method 1: Use the Command-Line Flag

The simplest way to override the default thread limit for a single scan is the `-max-threads` CLI argument defined in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) at line 115.

```bash
python sf.py -max-threads 20 -s example.com

```

This runs the scan with up to **20 modules executing simultaneously**, regardless of the default configuration.

## Method 2: Modify the Default Configuration in sf.py

The built-in default is stored in the `sfConfig` dictionary inside [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) at line 56.

```python

# sf.py - default configuration

sfConfig = {
    # ... other options ...

    '_maxthreads': 3,   # Default: 3 concurrent threads

    # ...

}

```

Change this value to set a new default for all future scans run from this installation.

## Method 3: Set _maxthreads in a Custom Configuration File

For reusable, environment-specific settings, create or edit a configuration file:

```ini

# sf.cfg

_maxthreads = 15

```

Then reference it when launching:

```bash
python sf.py -c sf.cfg -s example.com

```

SpiderFoot loads this file and applies the `_maxthreads` value throughout the scan lifecycle.

## How the Thread Limit Is Enforced Internally

The actual thread pool creation occurs in [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py) at line 213, where the `_maxthreads` value from your selected configuration method is passed to `SpiderFootThreadPool`:

```python

# sfscan.py – thread pool instantiation

self.__sharedThreadPool = SpiderFootThreadPool(
    threads=self.__config.get("_maxthreads", 3),
    name='sharedThreadPool')

```

All modules that support threading submit work to this shared instance, meaning the `_maxthreads` value caps **total concurrent activity across the entire scan**, not per-module behavior.

## Programmatic Configuration Example

When embedding SpiderFoot in another Python script, adjust the thread count directly:

```python
import sf

# Load default configuration and override

cfg = sf.sfConfig.copy()
cfg['_maxthreads'] = 25

# Initialize and start scan with modified config

scanner = sf.sfScan(target='example.com', sfConfig=cfg)
scanner.start()

```

## Key Source Files

| File | Purpose |
|------|---------|
| [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) | Defines `_maxthreads` default, CLI flag (`-max-threads`), and configuration schema |
| [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py) | Creates the shared `SpiderFootThreadPool` using the configured thread count |
| [`threadpool.py`](https://github.com/smicallef/spiderfoot/blob/main/threadpool.py) | Implements `SpiderFootThreadPool` class that manages worker threads and enforces the limit |

## Summary

- **Primary control**: The `_maxthreads` configuration option governs SpiderFoot's module concurrency
- **Three configuration methods**: CLI flag (`-max-threads`), [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) default dictionary, or external config file
- **Implementation location**: [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py) line 213 initializes the thread pool with your specified value
- **Scope**: The limit applies globally to all threaded modules within a single scan

## Frequently Asked Questions

### What is the default maximum concurrent threads in SpiderFoot?

The default value is **3 threads**, defined in the `sfConfig` dictionary at [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) line 56. This conservative default minimizes resource impact on standard hardware.

### Does increasing _maxthreads always improve scan speed?

Not necessarily. Higher values increase CPU and memory utilization and may trigger rate limiting from target services. The optimal setting depends on network bandwidth, target sensitivity, and local system capacity.

### Can different scans use different thread limits simultaneously?

Yes. Each `sfScan` instance creates its own `SpiderFootThreadPool` with the thread count specified in its configuration. Concurrent scans with different `_maxthreads` values operate independently.

### Where is the thread pool actually implemented?

The `SpiderFootThreadPool` class resides in [`threadpool.py`](https://github.com/smicallef/spiderfoot/blob/main/threadpool.py), which provides the underlying thread management and task queueing mechanisms used by [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py).