# SpiderFoot SQLite Database Schema: A Complete Technical Guide

> Explore the SpiderFoot SQLite database schema with this technical guide. Understand its eight core tables and indexes defined in spiderfootdb.py for efficient data management.

- Repository: [Steve Micallef/spiderfoot](https://github.com/smicallef/spiderfoot)
- Tags: deep-dive
- Published: 2026-08-15

---

**SpiderFoot stores all reconnaissance data in a single SQLite file with eight core tables and multiple indexes, defined programmatically in [`spiderfoot/db.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/db.py).**

SpiderFoot, the open-source reconnaissance and footprinting tool, persists all scan data to a local SQLite database. Understanding the **SpiderFoot SQLite database schema** is essential for anyone building custom queries, developing integrations, or troubleshooting scan issues. This guide breaks down every table, relationship, and optimization based on the actual source code in the smicallef/spiderfoot repository.

## Core Schema Overview

The schema is created dynamically via the `createSchemaQueries` list in **[`spiderfoot/db.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/db.py)** (lines 40-100). Rather than using a static SQL file, SpiderFoot builds tables programmatically, allowing for version-aware migrations and flexible deployment.

All data lives in eight interconnected tables with foreign key relationships that enforce referential integrity between scans, results, and events.

## The Eight Database Tables

### 1. tbl_event_types — Master Event Definitions

The foundation of SpiderFoot's data model. Every possible finding type is registered here before use.

- **Primary key:** `event` (VARCHAR)
- **Key columns:** `event_descr`, `event_raw` (boolean flag), `event_type` (category)

This table powers the `SpiderFootDb.eventTypes()` method, which returns the canonical list of supported indicators such as `IP_ADDRESS`, `DOMAIN_NAME`, `EMAILADDR`, and `DNS_TEXT`.

### 2. tbl_config — Global and Module Settings

Stores key-value pairs for SpiderFoot's configuration system.

- **Composite primary key:** (`scope`, `opt`)
- **Columns:** `scope` (global or module name), `opt` (option name), `val` (string value)

Used for persistent settings across restarts, separate from scan-specific overrides.

### 3. tbl_scan_instance — Scan Run Metadata

One row per reconnaissance job. This is the central anchor table referenced by nearly all other entities.

| Column | Purpose |
|--------|---------|
| `guid` | Unique scan identifier (primary key) |
| `name` | Human-readable scan label |
| `seed_target` | Original target (domain, IP, etc.) |
| `created`/`started`/`ended` | Unix timestamps for lifecycle tracking |
| `status` | Current state: `CREATED`, `STARTING`, `RUNNING`, `ABORT-REQUESTED`, `ABORTED`, `FINISHED`, `ERROR` |

The `scanInstanceCreate()` method in [`db.py`](https://github.com/smicallef/spiderfoot/blob/main/db.py) populates this table when the SpiderFoot engine initializes a new investigation.

### 4. tbl_scan_log — Runtime Diagnostics

Captures information, warning, and error messages generated during scans.

- **Foreign key:** `scan_instance_id` → `tbl_scan_instance(guid)`
- **Columns:** `generated` (timestamp), `component` (module name), `type` (log level), `message`

Essential for debugging module behavior and audit trails. No primary key is defined, allowing unlimited log entries per scan.

### 5. tbl_scan_config — Per-Scan Overrides

Module-specific settings that apply only to a single scan instance, overriding global `tbl_config` values.

- **Foreign key:** `scan_instance_id` → `tbl_scan_instance(guid)`
- **Columns:** `component`, `opt`, `val`

Enables the same module to run with different parameters across concurrent scans.

### 6. tbl_scan_results — The Core Findings Table

The workhorse of the schema. Every discovered asset, relationship, or indicator lands here.

| Column | Type/Constraint | Purpose |
|--------|---------------|---------|
| `scan_instance_id` | FK → `tbl_scan_instance` | Parent scan |
| `hash` | VARCHAR NOT NULL | Unique content hash (prevents duplicates) |
| `type` | FK → `tbl_event_types(event)` | Event classification |
| `generated` | INT NOT NULL | Discovery timestamp |
| `confidence` | INT DEFAULT 100 | Reliability score (0-100) |
| `visibility` | INT DEFAULT 100 | Visibility score (0-100) |
| `risk` | INT DEFAULT 0 | Risk score (0-10 typically) |
| `module` | VARCHAR NOT NULL | Source module name |
| `data` | VARCHAR | The actual finding content |
| `false_positive` | INT DEFAULT 0 | Manual FP flag |
| `source_event_hash` | VARCHAR DEFAULT 'ROOT' | Parent finding in chain |

The `hash` field enables deduplication: identical findings from different modules within the same scan are collapsed to a single row. The `source_event_hash` creates parent-child relationships, building investigation trees.

### 7. tbl_scan_correlation_results — Rule Engine Outputs

Stores matches from SpiderFoot's correlation engine, which identifies patterns across multiple findings.

- **Primary key:** `id` (correlation result UUID)
- **Foreign key:** `scan_instance_id` → `tbl_scan_instance(guid)`
- **Rule metadata:** `title`, `rule_risk`, `rule_id`, `rule_name`, `rule_descr`, `rule_logic`

Correlation rules express complex conditions like "flag when an IP address is associated with both a malicious domain and a suspicious SSL certificate."

### 8. tbl_scan_correlation_results_events — Join Table

Links correlation results to the specific findings that triggered them.

- **Foreign keys:** `correlation_id` → `tbl_scan_correlation_results(id)`, `event_hash` → `tbl_scan_results(hash)`

This many-to-many relationship enables reconstruction of exactly which data points satisfied a correlation rule's logic.

## Performance Indexes

SpiderFoot creates eight targeted indexes in **[`spiderfoot/db.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/db.py)** (lines 101-108) to accelerate common query patterns:

- `idx_scan_results_id` — Fast lookup of all results for a scan
- `idx_scan_results_type` — Filter by event type within a scan
- `idx_scan_results_hash` — Deduplication checks and exact match retrieval
- `idx_scan_results_module` — Analyze single module contributions
- `idx_scan_results_srchash` — Traverse result dependency chains
- `idx_scan_logs` — Retrieve scan diagnostics efficiently
- `idx_scan_correlation` — List correlations for a scan
- `idx_scan_correlation_events` — Resolve correlation result membership

## Working with the Schema in Python

The `SpiderFootDb` class in **[`spiderfoot/db.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/db.py)** provides high-level methods that abstract raw SQL. Common operations:

```python
from spiderfoot.db import SpiderFootDb

# Initialize database (creates tables if missing)

opts = {'__database': '/opt/spiderfoot/spiderfoot.db'}
db = SpiderFootDb(opts, init=True)

# Query supported event types

event_types = db.eventTypes()

# Returns: [('Description', 'EVENT_CODE', is_raw, 'category'), ...]

# Create scan programmatically

db.scanInstanceCreate(
    '550e8400-e29b-41d4-a716-446655440000',
    'Asset Discovery: example.com',
    'example.com'
)

# Search with filters

dns_txt = db.search({
    'scan_id': '550e8400-e29b-41d4-a716-446655440000',
    'type': 'DNS_TEXT'
})

# Aggregate statistics

summary = db.scanResultSummary(
    '550e8400-e29b-41d4-a716-446655440000',
    by='entity'
)

# Retrieve correlation alerts

alerts = db.scanCorrelationList('550e8400-e29b-41d4-a716-446655440000')

```

For raw SQL access, the database path is available via the `__database` configuration key and can be opened with any SQLite client.

## File Locations and Architecture

| File | Responsibility |
|------|--------------|
| **[`spiderfoot/db.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/db.py)** | Schema definition, `SpiderFootDb` class, all CRUD operations |
| **[`spiderfoot/event.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/event.py)** | `SpiderFootEvent` class — in-memory representation that maps to `tbl_event_types` |
| **[`spiderfoot/scan.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/scan.py)** | Scan orchestration that populates tables via `SpiderFootDb` methods |

The persistence layer is entirely self-contained in [`db.py`](https://github.com/smicallef/spiderfoot/blob/main/db.py). No ORM is used — SQL is constructed directly for predictable performance and minimal dependencies.

## Summary

- **SpiderFoot SQLite database schema** consists of eight tables: `tbl_event_types`, `tbl_config`, `tbl_scan_instance`, `tbl_scan_log`, `tbl_scan_config`, `tbl_scan_results`, `tbl_scan_correlation_results`, and `tbl_scan_correlation_results_events`
- Schema creation is programmatic in [`spiderfoot/db.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/db.py) via `createSchemaQueries`
- Foreign key relationships enforce integrity between scans, results, and correlations
- The `hash` column in `tbl_scan_results` enables finding deduplication
- Eight indexes optimize the most common query patterns
- `SpiderFootDb` class provides Pythonic access without requiring raw SQL

## Frequently Asked Questions

### Where is the SpiderFoot database file located?

The database path is controlled by the `__database` configuration option, defaulting to `spiderfoot.db` in the application root. Specify a custom path during initialization: `SpiderFootDb({'__database': '/custom/path.db'}, init=True)`.

### Can I query the SpiderFoot database with external tools?

Yes. Since SpiderFoot uses standard SQLite, you can open the `.db` file with `sqlite3` CLI, DBeaver, or Python's `sqlite3` module. Be aware that SpiderFoot maintains long-running locks during active scans.

### What is the difference between `tbl_config` and `tbl_scan_config`?

`tbl_config` stores global and default module settings. `tbl_scan_config` captures overrides for specific scan instances, enabling the same module to run with different parameters across concurrent investigations. Scan config values take precedence at runtime.

### How does SpiderFoot prevent duplicate results?

The `hash` column in `tbl_scan_results` is computed from the finding's content and type. Before insertion, SpiderFoot checks for existing matching hashes within the same scan instance, suppressing redundant entries from multiple modules.