SpiderFoot SQLite Database Schema: A Complete Technical Guide

SpiderFoot stores all reconnaissance data in a single SQLite file with eight core tables and multiple indexes, defined programmatically in spiderfoot/db.py.

SpiderFoot, the open-source reconnaissance and footprinting tool, persists all scan data to a local SQLite database. Understanding the SpiderFoot SQLite database schema is essential for anyone building custom queries, developing integrations, or troubleshooting scan issues. This guide breaks down every table, relationship, and optimization based on the actual source code in the smicallef/spiderfoot repository.

Core Schema Overview

The schema is created dynamically via the createSchemaQueries list in spiderfoot/db.py (lines 40-100). Rather than using a static SQL file, SpiderFoot builds tables programmatically, allowing for version-aware migrations and flexible deployment.

All data lives in eight interconnected tables with foreign key relationships that enforce referential integrity between scans, results, and events.

The Eight Database Tables

1. tbl_event_types — Master Event Definitions

The foundation of SpiderFoot's data model. Every possible finding type is registered here before use.

• Primary key: event (VARCHAR)
• Key columns: event_descr, event_raw (boolean flag), event_type (category)

This table powers the SpiderFootDb.eventTypes() method, which returns the canonical list of supported indicators such as IP_ADDRESS, DOMAIN_NAME, EMAILADDR, and DNS_TEXT.

2. tbl_config — Global and Module Settings

Stores key-value pairs for SpiderFoot's configuration system.

• Composite primary key: (scope, opt)
• Columns: scope (global or module name), opt (option name), val (string value)

Used for persistent settings across restarts, separate from scan-specific overrides.

3. tbl_scan_instance — Scan Run Metadata

One row per reconnaissance job. This is the central anchor table referenced by nearly all other entities.

Column	Purpose
guid	Unique scan identifier (primary key)
name	Human-readable scan label
seed_target	Original target (domain, IP, etc.)
created/started/ended	Unix timestamps for lifecycle tracking
status	Current state: CREATED, STARTING, RUNNING, ABORT-REQUESTED, ABORTED, FINISHED, ERROR

The scanInstanceCreate() method in db.py populates this table when the SpiderFoot engine initializes a new investigation.

4. tbl_scan_log — Runtime Diagnostics

Captures information, warning, and error messages generated during scans.

• Foreign key: scan_instance_id → tbl_scan_instance(guid)
• Columns: generated (timestamp), component (module name), type (log level), message

Essential for debugging module behavior and audit trails. No primary key is defined, allowing unlimited log entries per scan.

5. tbl_scan_config — Per-Scan Overrides

Module-specific settings that apply only to a single scan instance, overriding global tbl_config values.

• Foreign key: scan_instance_id → tbl_scan_instance(guid)
• Columns: component, opt, val

Enables the same module to run with different parameters across concurrent scans.

6. tbl_scan_results — The Core Findings Table

The workhorse of the schema. Every discovered asset, relationship, or indicator lands here.

Column	Type/Constraint	Purpose
scan_instance_id	FK → tbl_scan_instance	Parent scan
hash	VARCHAR NOT NULL	Unique content hash (prevents duplicates)
type	FK → tbl_event_types(event)	Event classification
generated	INT NOT NULL	Discovery timestamp
confidence	INT DEFAULT 100	Reliability score (0-100)
visibility	INT DEFAULT 100	Visibility score (0-100)
risk	INT DEFAULT 0	Risk score (0-10 typically)
module	VARCHAR NOT NULL	Source module name
data	VARCHAR	The actual finding content
false_positive	INT DEFAULT 0	Manual FP flag
source_event_hash	VARCHAR DEFAULT 'ROOT'	Parent finding in chain

The hash field enables deduplication: identical findings from different modules within the same scan are collapsed to a single row. The source_event_hash creates parent-child relationships, building investigation trees.

7. tbl_scan_correlation_results — Rule Engine Outputs

Stores matches from SpiderFoot's correlation engine, which identifies patterns across multiple findings.

• Primary key: id (correlation result UUID)
• Foreign key: scan_instance_id → tbl_scan_instance(guid)
• Rule metadata: title, rule_risk, rule_id, rule_name, rule_descr, rule_logic

Correlation rules express complex conditions like "flag when an IP address is associated with both a malicious domain and a suspicious SSL certificate."

8. tbl_scan_correlation_results_events — Join Table

Links correlation results to the specific findings that triggered them.

• Foreign keys: correlation_id → tbl_scan_correlation_results(id), event_hash → tbl_scan_results(hash)

This many-to-many relationship enables reconstruction of exactly which data points satisfied a correlation rule's logic.

Performance Indexes

SpiderFoot creates eight targeted indexes in spiderfoot/db.py (lines 101-108) to accelerate common query patterns:

• idx_scan_results_id — Fast lookup of all results for a scan
• idx_scan_results_type — Filter by event type within a scan
• idx_scan_results_hash — Deduplication checks and exact match retrieval
• idx_scan_results_module — Analyze single module contributions
• idx_scan_results_srchash — Traverse result dependency chains
• idx_scan_logs — Retrieve scan diagnostics efficiently
• idx_scan_correlation — List correlations for a scan
• idx_scan_correlation_events — Resolve correlation result membership

Working with the Schema in Python

The SpiderFootDb class in spiderfoot/db.py provides high-level methods that abstract raw SQL. Common operations:

from spiderfoot.db import SpiderFootDb

# Initialize database (creates tables if missing)

opts = {'__database': '/opt/spiderfoot/spiderfoot.db'}
db = SpiderFootDb(opts, init=True)

# Query supported event types

event_types = db.eventTypes()

# Returns: [('Description', 'EVENT_CODE', is_raw, 'category'), ...]

# Create scan programmatically

db.scanInstanceCreate(
    '550e8400-e29b-41d4-a716-446655440000',
    'Asset Discovery: example.com',
    'example.com'
)

# Search with filters

dns_txt = db.search({
    'scan_id': '550e8400-e29b-41d4-a716-446655440000',
    'type': 'DNS_TEXT'
})

# Aggregate statistics

summary = db.scanResultSummary(
    '550e8400-e29b-41d4-a716-446655440000',
    by='entity'
)

# Retrieve correlation alerts

alerts = db.scanCorrelationList('550e8400-e29b-41d4-a716-446655440000')

For raw SQL access, the database path is available via the __database configuration key and can be opened with any SQLite client.

File Locations and Architecture

File	Responsibility
spiderfoot/db.py	Schema definition, SpiderFootDb class, all CRUD operations
spiderfoot/event.py	SpiderFootEvent class — in-memory representation that maps to tbl_event_types
spiderfoot/scan.py	Scan orchestration that populates tables via SpiderFootDb methods

The persistence layer is entirely self-contained in db.py. No ORM is used — SQL is constructed directly for predictable performance and minimal dependencies.

Summary

• SpiderFoot SQLite database schema consists of eight tables: tbl_event_types, tbl_config, tbl_scan_instance, tbl_scan_log, tbl_scan_config, tbl_scan_results, tbl_scan_correlation_results, and tbl_scan_correlation_results_events
• Schema creation is programmatic in spiderfoot/db.py via createSchemaQueries
• Foreign key relationships enforce integrity between scans, results, and correlations
• The hash column in tbl_scan_results enables finding deduplication
• Eight indexes optimize the most common query patterns
• SpiderFootDb class provides Pythonic access without requiring raw SQL

Frequently Asked Questions

Where is the SpiderFoot database file located?

The database path is controlled by the __database configuration option, defaulting to spiderfoot.db in the application root. Specify a custom path during initialization: SpiderFootDb({'__database': '/custom/path.db'}, init=True).

Can I query the SpiderFoot database with external tools?

Yes. Since SpiderFoot uses standard SQLite, you can open the .db file with sqlite3 CLI, DBeaver, or Python's sqlite3 module. Be aware that SpiderFoot maintains long-running locks during active scans.

What is the difference between tbl_config and tbl_scan_config?

tbl_config stores global and default module settings. tbl_scan_config captures overrides for specific scan instances, enabling the same module to run with different parameters across concurrent investigations. Scan config values take precedence at runtime.

How does SpiderFoot prevent duplicate results?

The hash column in tbl_scan_results is computed from the finding's content and type. Before insertion, SpiderFoot checks for existing matching hashes within the same scan instance, suppressing redundant entries from multiple modules.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}