# spiderfoot | Steve Micallef | Knowledge Base | Instagit SpiderFoot automates OSINT for threat intelligence and mapping your attack surface. GitHub Stars: 19k Repository: https://github.com/smicallef/spiderfoot --- ## Articles ### [How to Troubleshoot Database Locking and Concurrency Issues in SpiderFoot](/smicallef/spiderfoot/how-to-troubleshoot-database-locking-and-concurrency-issues-in-spiderfoot) Troubleshoot SpiderFoot database locking and concurrency issues. Learn to enable WAL mode, use retry loops, adjust thread pools, and prevent external locks for smooth operation. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Optimize SpiderFoot Scan Performance for Large Targets: Thread Pool Tuning and DNS Optimization Guide](/smicallef/spiderfoot/how-to-optimize-spiderfoot-scan-performance-for-large-targets) Boost SpiderFoot scan performance on large targets by tuning thread pools and optimizing DNS. Eliminate bottlenecks and speed up your scans with this essential guide. - Tags: performance - Published: 2026-08-15 ### [How to Monitor SpiderFoot Scan Progress Programmatically via Its REST API](/smicallef/spiderfoot/how-to-monitor-spiderfoot-scan-progress-programmatically-via-its-rest-api) Monitor SpiderFoot scan progress programmatically using the REST API. Poll the scanopts endpoint for real-time status and metadata without database access. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Customize User Agent Strings for HTTP Requests in SpiderFoot](/smicallef/spiderfoot/how-to-customize-user-agent-strings-for-http-requests-in-spiderfoot) Learn how to customize User Agent strings for SpiderFoot HTTP requests. Override default agents globally with static strings, local files, or remote URLs. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Create and Test Custom Correlation Rules in SpiderFoot](/smicallef/spiderfoot/how-to-create-and-test-custom-correlation-rules-in-spiderfoot) Learn to create and test custom correlation rules in SpiderFoot. This guide explains YAML file creation, loading, execution, and verification for powerful threat intelligence. - Tags: how-to-guide - Published: 2026-08-15 ### [How the SpiderFoot Web UI Communicates with the Backend Scan Engine: Architecture Deep Dive](/smicallef/spiderfoot/how-does-the-spiderfoot-web-ui-communicate-with-the-backend-scan-engine) Discover how SpiderFoot's web UI interacts with its backend scan engine. Explore the architecture using CherryPy HTTP endpoints, multiprocessing, and SQLite for seamless coordination. - Tags: architecture - Published: 2026-08-15 ### [How to Handle API Key Errors and Authentication Failures in SpiderFoot Modules](/smicallef/spiderfoot/how-to-handle-api-key-errors-and-authentication-failures-in-spiderfoot-modules) Learn to handle API key errors and authentication failures in SpiderFoot modules. Discover the standardized error-handling pattern to prevent redundant requests and ensure smooth data collection. - Tags: how-to-guide - Published: 2026-08-15 ### [Best Practices for Scanning Sensitive Targets Responsively with SpiderFoot: A Complete Guide](/smicallef/spiderfoot/what-are-best-practices-for-scanning-sensitive-targets-responsibly-with-spiderfoot) Learn best practices for scanning sensitive targets with SpiderFoot. Get explicit permission, limit scope, use safe mode, enforce rate limits, and protect data. - Tags: best-practices - Published: 2026-08-15 ### [How SpiderFoot's Thread Pool Manages Parallel Module Execution](/smicallef/spiderfoot/how-does-spiderfoot-s-thread-pool-manage-parallel-module-execution) Discover how SpiderFoot's thread pool efficiently manages parallel module execution using worker threads, named queues, and configurable concurrency for faster threat intelligence gathering. - Tags: internals - Published: 2026-08-15 ### [How to Integrate External Tools Like Nmap, DNSTwist, and WhatWeb with SpiderFoot](/smicallef/spiderfoot/how-to-integrate-external-tools-like-nmap-dnstwist-and-whatweb-with-spiderfoot) Learn how to integrate external tools like Nmap, DNSTwist, and WhatWeb with SpiderFoot. Enhance your reconnaissance with SpiderFoot's powerful plugin architecture. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot Performs DNS Resolution and Zone Transfers: A Technical Deep Dive](/smicallef/spiderfoot/how-does-spiderfoot-perform-dns-resolution-and-zone-transfers) Explore how SpiderFoot performs DNS resolution and zone transfers using dedicated plugins and core library methods. Uncover the technical details of its DNS operations. - Tags: deep-dive - Published: 2026-08-15 ### [How to Add Custom Threat Intelligence Feeds to SpiderFoot: A Complete Guide](/smicallef/spiderfoot/how-to-add-custom-threat-intelligence-feeds-to-spiderfoot) Easily integrate custom threat intelligence feeds into SpiderFoot using the sfp_customfeed module. Import JSON indicator lists without code changes for enhanced security. - Tags: how-to-guide - Published: 2026-08-15 ### [How Different SpiderFoot Target Types (IP, Domain, Email, ASN) Affect Scan Behavior](/smicallef/spiderfoot/how-do-different-spiderfoot-target-types-ip-domain-email-asn-affect-scan-behavior) Discover how SpiderFoot target types IP, domain, email, and ASN influence scan behavior, module selection, and data mapping. Optimize your threat intelligence gathering. - Tags: deep-dive - Published: 2026-08-15 ### [How to Configure the Maximum Concurrent Threads for SpiderFoot Module Execution](/smicallef/spiderfoot/how-to-configure-the-maximum-concurrent-threads-for-spiderfoot-module-execution) Learn how to configure maximum concurrent threads for SpiderFoot module execution using CLI flags, config files, or the default config dictionary for optimized performance. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Export SpiderFoot Scan Results to CSV, JSON, or GEXF Formats](/smicallef/spiderfoot/how-to-export-spiderfoot-scan-results-to-csv-json-or-gexf-formats) Easily export SpiderFoot scan results to CSV, JSON, or GEXF. Learn how to access your intelligence data using web UI, CLI, or HTTP requests. - Tags: how-to-guide - Published: 2026-08-15 ### [SpiderFoot SQLite Database Schema: A Complete Technical Guide](/smicallef/spiderfoot/what-is-the-structure-of-spiderfoot-s-sqlite-database-schema) Explore the SpiderFoot SQLite database schema with this technical guide. Understand its eight core tables and indexes defined in spiderfootdb.py for efficient data management. - Tags: deep-dive - Published: 2026-08-15 ### [How to Use SpiderFoot Exclusively via the Command-Line Interface (CLI)](/smicallef/spiderfoot/how-to-use-spiderfoot-exclusively-via-the-command-line-interface-cli) Master SpiderFoot via the command-line interface (CLI). Control scans, retrieve data, and export results without the web UI. Learn how to use sf.py and sfcli.py for full automation. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot Implements Rate Limiting Across Its Modules: A Deep Dive into the Thread-Pool Architecture](/smicallef/spiderfoot/how-does-spiderfoot-implement-rate-limiting-across-its-modules) Learn how SpiderFoot implements rate limiting using a thread-pool architecture and explicit 429 error handling to manage module requests effectively. - Tags: deep-dive - Published: 2026-08-15 ### [How to Debug Module Failures and Handle Errors in SpiderFoot](/smicallef/spiderfoot/how-to-debug-module-failures-and-handle-errors-in-spiderfoot) Debug SpiderFoot module failures and handle errors effectively. Learn to enable verbose logging, check logs for stack traces, and validate module configurations for seamless operation. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot’s YAML‑Based Correlation Engine Processes Scan Results](/smicallef/spiderfoot/how-does-spiderfoot-s-yaml-based-correlation-engine-process-scan-results) Learn how SpiderFoot's YAML-based correlation engine processes scan results by loading rules querying scan data enriching events and analyzing patterns for actionable intelligence. - Tags: internals - Published: 2026-08-15 ### [How to Write a Custom SpiderFoot Module in Python: A Step-by-Step Guide](/smicallef/spiderfoot/how-to-write-a-custom-spiderfoot-module-in-python) Learn to write a custom SpiderFoot module in Python with this step-by-step guide. Extend SpiderFoot's capabilities by creating your own modules and integrating new data sources. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot Modules Communicate Using the Pub/Sub Event System](/smicallef/spiderfoot/how-do-spiderfoot-modules-communicate-using-the-pub-sub-event-system) Discover how SpiderFoot modules communicate via a pub/sub system. Learn about consumes, provides, notifyListeners, and handleEvent for efficient data exchange. - Tags: internals - Published: 2026-08-15 ### [How to Configure SOCKS5/TOR Proxy for Anonymized Scanning in SpiderFoot](/smicallef/spiderfoot/how-to-configure-socks5-tor-proxy-for-anonymized-scanning-in-spiderfoot) Learn to configure SOCKS5/TOR proxy for anonymized scanning in SpiderFoot. Enhance your OSINT investigations with secure, private data gathering. Step-by-step guide included. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot's Module Plugin Architecture Works: A Deep Dive into the Plugin Framework](/smicallef/spiderfoot/how-does-spiderfoot-s-module-plugin-architecture-work) Explore SpiderFoot's module plugin architecture. Learn how interchangeable data-source modules communicate via an event-driven system built on the SpiderFootPlugin base class. - Tags: deep-dive - Published: 2026-08-15 ### [How to Create Custom Correlation Rules in SpiderFoot Using YAML](/smicallef/spiderfoot/create-custom-correlation-rules-spiderfoot-yaml) Learn to create custom correlation rules in SpiderFoot using YAML. Discover relationships between data elements with this straightforward guide. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot Handles Data Storage and Database Schema in SQLite](/smicallef/spiderfoot/spiderfoot-data-storage-database-schema) Discover how SpiderFoot manages data storage using SQLite and its database schema. Learn about thread-safe wrappers, schema creation, event logging, and scan results. - Tags: internals - Published: 2026-08-15 ### [How to Implement Subnet and ASN Scanning in SpiderFoot: A Complete Technical Guide](/smicallef/spiderfoot/implement-subnet-asn-scanning-spiderfoot) Learn to implement subnet and ASN scanning in SpiderFoot with this technical guide. Discover how to leverage its components and modules for comprehensive network reconnaissance. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Troubleshoot SpiderFoot Module Failures and Debug Them: Complete Guide](/smicallef/spiderfoot/troubleshoot-spiderfoot-module-failures-debugging) Fix SpiderFoot module failures by enabling debug logging and examining logs. Learn to identify and resolve syntax errors, API issues, and timeout problems for efficient debugging. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Use SpiderFoot's Built-In Data Visualizations: A Complete Guide for Security Analysts](/smicallef/spiderfoot/use-spiderfoot-built-in-visualizations) Unlock SpiderFoot's built-in data visualizations. Learn how security analysts can leverage interactive D3.js charts to understand scan relationships and enhance investigations through our complete guide. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Filter Event Types During a SpiderFoot Scan: A Complete Guide](/smicallef/spiderfoot/filter-event-types-spiderfoot-scan) Learn to filter event types in SpiderFoot scans using the /scaneventresults API. Dynamically build SQL WHERE clauses to retrieve only the events you need. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot's DNS Resolution Customization Works: A Complete Guide to Using Custom DNS Servers](/smicallef/spiderfoot/spiderfoot-dns-resolution-customization) Learn how to customize SpiderFoot's DNS resolution using custom DNS servers. Override system defaults for all dnspython and socket lookups with this complete guide. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Integrate Nmap, DNSTwist, and WhatWeb with SpiderFoot: Complete Plugin Integration Guide](/smicallef/spiderfoot/integrate-external-tools-spiderfoot) Learn to integrate Nmap, DNSTwist, and WhatWeb with SpiderFoot using its plugin architecture. Enhance your OSINT investigations with this comprehensive guide. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Customize the User Agent String for SpiderFoot HTTP Requests](/smicallef/spiderfoot/customize-user-agent-spiderfoot-http-requests) Learn to customize the SpiderFoot User-Agent string for HTTP requests. Override globally with a string, local file, or remote URL for flexible scraping. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot Handles Rate Limiting and Request Throttling: A Deep Dive into Its 3-Layer Architecture](/smicallef/spiderfoot/spiderfoot-rate-limiting-request-throttling) Learn how SpiderFoot expertly manages rate limiting with its 3-layer architecture. Discover its thread-pool concurrency, module delays, and error handling to avoid API bans. - Tags: deep-dive - Published: 2026-08-15 ### [How to Use SpiderFoot with Docker and Docker-Compose: Complete Deployment Guide](/smicallef/spiderfoot/use-spiderfoot-docker-docker-compose) Deploy SpiderFoot with Docker and docker-compose for a production-ready OSINT platform. Our guide simplifies setup, ensures isolated environments, and enables persistent storage for your investigations. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Enable and Configure Logging in SpiderFoot: Complete Developer Guide](/smicallef/spiderfoot/enable-configure-logging-spiderfoot) Learn to enable and configure logging in SpiderFoot. This guide details using Python's logging module with SQLite for efficient scan activity and error capture. Master SpiderFoot logs for better insights. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot Handles Different Target Types: Domain, IP, Email, and More](/smicallef/spiderfoot/spiderfoot-target-type-handling-explained) Learn how SpiderFoot handles diverse target types like domains, IPs, and emails. Explore its flexible matching logic for effective OSINT investigations. - Tags: deep-dive - Published: 2026-08-15 ### [How to Configure API Keys for Premium SpiderFoot Modules: Shodan and HaveIBeenPwned Setup Guide](/smicallef/spiderfoot/configure-api-keys-spiderfoot-premium-modules) Easily configure API keys for premium SpiderFoot modules like Shodan and HaveIBeenPwned. Follow our guide to unlock advanced recon capabilities and enhance your threat intelligence. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Use Strict Mode (`-x` Flag) in the SpiderFoot CLI](/smicallef/spiderfoot/use-spiderfoot-cli-strict-mode-x-flag) Learn to use SpiderFoot CLI strict mode with the -x flag. Limit scans to relevant modules and collect specific event types for focused reconnaissance. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Export SpiderFoot Scan Results to CSV, JSON, and GEXF Formats](/smicallef/spiderfoot/export-spiderfoot-scan-results-csv-json-gexf) Export SpiderFoot scan results to CSV JSON and GEXF formats using the web UI CLI or API. Learn how to convert your data efficiently for analysis. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot Handles Module Concurrency with Its Thread Pool](/smicallef/spiderfoot/spiderfoot-module-concurrency-thread-pool) Discover how SpiderFoot handles module concurrency using its thread pool. Learn about named queues, thread limits, and shared worker threads for efficient reconnaissance. - Tags: internals - Published: 2026-08-15 ### [How to Run SpiderFoot in CLI Mode with Specific Modules and Targets](/smicallef/spiderfoot/run-spiderfoot-cli-mode-modules-targets) Learn to run SpiderFoot in CLI mode. Control specific modules and targets with sfcli.py for precise, targeted scans via the REST API. Master automated OSINT efficiently. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Configure SpiderFoot's Web UI with Authentication and CORS Settings](/smicallef/spiderfoot/configure-spiderfoot-web-ui-authentication-cors) Secure your SpiderFoot web UI with HTTP Digest authentication and enable CORS by following these configuration steps. Learn to set up password files and add response headers for enhanced security. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Use SpiderFoot's SQLite Database for Custom Queries: A Complete Guide](/smicallef/spiderfoot/use-spiderfoot-sqlite-database-custom-queries) Learn to leverage SpiderFoot's SQLite database for custom queries. Access scan data programmatically via Python or directly with SQL for in-depth analysis. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot's Correlation Engine Works with YAML Rules: A Complete Technical Guide](/smicallef/spiderfoot/how-spiderfoot-correlation-engine-yaml-rules) Discover how SpiderFoot's correlation engine uses YAML rules to automate OSINT analysis, detect relationships, and generate actionable security intelligence. Learn the technical details. - Tags: deep-dive - Published: 2026-08-15 ### [How to Write a Custom SpiderFoot Module in Python: A Complete Developer Guide](/smicallef/spiderfoot/write-custom-spiderfoot-module-python) Learn to write a custom SpiderFoot module in Python. This guide covers inheriting from SpiderFootPlugin, defining metadata, handling events, and producing findings for enhanced threat intelligence. - Tags: how-to-guide - Published: 2026-08-15 ### [How to Configure SOCKS5 and TOR Proxy Settings for SpiderFoot Scans](/smicallef/spiderfoot/configure-socks5-tor-proxy-spiderfoot-scans) Learn to configure SOCKS5 and TOR proxy settings for SpiderFoot scans. Enhance your OSINT by routing traffic through anonymity networks for more secure and comprehensive data collection. - Tags: how-to-guide - Published: 2026-08-15 ### [How SpiderFoot's Module Event System Works: A Deep Dive Into Its Publisher/Subscriber Model](/smicallef/spiderfoot/how-spiderfoot-module-event-system-publisher-subscriber-model) Explore SpiderFoot's module event system and its pub/sub model. Learn how events are emitted, persisted, and broadcast to listeners for efficient data processing. - Tags: deep-dive - Published: 2026-08-15 ### [How to Customize SpiderFoot's Web UI: Templates and Static Assets Guide](/smicallef/spiderfoot/customizing-spiderfoot-web-ui-templates-and-static-assets) Easily customize SpiderFoot's web UI by editing Mako templates and static assets. Learn how to modify the interface for your specific needs and enhance your OSINT workflow. - Tags: how-to-guide - Published: 2026-08-14 ### [SpiderFoot Error Handling and Retry Mechanisms: A Deep Dive into Resilient OSINT Scanning](/smicallef/spiderfoot/spiderfoot-error-handling-and-retry-mechanisms) Master SpiderFoot's error handling and retry mechanisms to build resilient OSINT scans. Learn about its three-tier architecture for managing network failures and API limits. - Tags: deep-dive - Published: 2026-08-14 ### [How to Disable or Filter Specific Modules During a Scan in SpiderFoot: 3 Methods Explained](/smicallef/spiderfoot/how-to-disable-or-filter-specific-modules-during-a-scan-in-spiderfoot) Learn 3 methods to disable or filter specific modules in SpiderFoot scans: Web UI, sfcli command-line, and REST API. Optimize your scans efficiently. - Tags: how-to-guide - Published: 2026-08-14 ### [Understanding SpiderFoot Event Types and the Event Hierarchy: A Complete Guide](/smicallef/spiderfoot/understanding-spiderfoot-event-types-and-the-event-hierarchy) Master SpiderFoot event types and hierarchy. Understand how SpiderFoot tracks data provenance with its event system and parent-child relationships for complete investigative insight. - Tags: deep-dive - Published: 2026-08-14 ### [Performance Tuning SpiderFoot for Large-Scale Scans: A Complete Guide](/smicallef/spiderfoot/performance-tuning-spiderfoot-for-large-scale-scans) Optimize SpiderFoot for massive scans. Learn to scale thread pools, tune module concurrency, and manage SQLite locks for peak performance. Get the complete guide. - Tags: performance - Published: 2026-08-14 ### [How to Integrate SpiderFoot with DNSTwist and WhatWeb: A Complete Plugin Guide](/smicallef/spiderfoot/integrating-spiderfoot-with-external-tools-like-dnstwist-and-whatweb) Learn how to integrate SpiderFoot with DNSTwist and WhatWeb. This guide details plugin usage for enhanced OSINT and data correlation. - Tags: how-to-guide - Published: 2026-08-14 ### [SpiderFoot Data Storage Architecture and Query Patterns Explained](/smicallef/spiderfoot/spiderfoot-data-storage-architecture-and-query-patterns) Explore SpiderFoot's data storage architecture and query patterns. Learn how intelligence is stored in SQLite, managed by the spiderfoot.db module, and optimized for efficient retrieval. - Tags: architecture - Published: 2026-08-14 ### [How to Debug and Log SpiderFoot Module Executions: Complete Guide with Code Examples](/smicallef/spiderfoot/how-to-debug-and-log-spiderfoot-module-executions) Debug SpiderFoot module executions efficiently. Enable debug mode, use self.debug() for structured logs, and send them to files, console, or SQLite database. Complete guide with code examples. - Tags: how-to-guide - Published: 2026-08-14 ### [Understanding SpiderFoot's Scan Lifecycle and State Machine: A Complete Developer Guide](/smicallef/spiderfoot/understanding-spiderfoot-scan-lifecycle-and-state-machine) Master SpiderFoot's scan lifecycle and state machine. Explore the seven immutable states INITIALIZING, STARTING, RUNNING, ABORTED, FINISHED, and ERROR-FAILED in this developer guide. - Tags: internals - Published: 2026-08-14 ### [Running SpiderFoot in Docker and Docker-Compose: Complete Deployment Guide](/smicallef/spiderfoot/running-spiderfoot-in-docker-and-docker-compose-configurations) Deploy SpiderFoot in Docker and Docker-Compose for a secure, isolated, and persistent OSINT solution. Follow our complete guide to get started. - Tags: how-to-guide - Published: 2026-08-14 ### [How SpiderFoot Handles Rate Limiting and Throttling Across Modules](/smicallef/spiderfoot/how-does-spiderfoot-handle-rate-limiting-and-throttling-across-modules) Discover how SpiderFoot uses thread-pool throttling and HTTP 429 detection to manage rate limiting and ensure smooth module operations when interacting with APIs. - Tags: internals - Published: 2026-08-14 ### [How SpiderFoot Module Dependencies Work and How Modules Feed Each Other](/smicallef/spiderfoot/spiderfoot-module-dependencies-and-how-modules-feed-each-other) Understand SpiderFoot module dependencies and how they feed each other. Learn how modules declare and consume events for automated data routing and enhanced intelligence gathering. - Tags: internals - Published: 2026-08-14 ### [How to Export SpiderFoot Scan Results to JSON, CSV, or GEXF Formats](/smicallef/spiderfoot/exporting-spiderfoot-scan-results-to-json-csv-or-gexf-formats) Easily export SpiderFoot scan results to JSON, CSV, or GEXF formats. Learn how to leverage SpiderFoot's export options for efficient data analysis and integration. - Tags: how-to-guide - Published: 2026-08-14 ### [Understanding SpiderFoot Target Types: Domains, IPs, Emails, Bitcoin Addresses, and More](/smicallef/spiderfoot/understanding-spiderfoot-target-types-domain-ip-email-btc-etc) Explore SpiderFoot target types including domains, IPs, emails, Bitcoin addresses, and more. Learn how SpiderFoot validates and matches diverse data for effective OSINT investigations. - Tags: deep-dive - Published: 2026-08-14 ### [How to Add API Keys to SpiderFoot Modules Securely: A Complete Guide](/smicallef/spiderfoot/how-to-add-api-keys-to-spiderfoot-modules-securely) Securely add API keys to SpiderFoot modules by using the central configuration system. Avoid hard-coding and enhance your data collection security. Get the complete guide. - Tags: how-to-guide - Published: 2026-08-14 ### [SpiderFoot ThreadPool Implementation and Concurrency Model: A Deep Dive into the Source Code](/smicallef/spiderfoot/what-is-spiderfoot-threadpool-implementation-and-concurrency-model) Explore SpiderFoot's custom threadpool implementation and concurrency model. Understand how worker threads, task queues, and dynamic throttling power efficient I/O-bound reconnaissance. - Tags: internals - Published: 2026-08-14 ### [Troubleshooting Module Failures and Errors in SpiderFoot Scans: A Complete Debugging Guide](/smicallef/spiderfoot/troubleshooting-module-failures-and-errors-in-spiderfoot-scans) Troubleshoot SpiderFoot module failures and errors with our complete debugging guide. Learn to enable debug logging, check logs, and query scan data to resolve issues. - Tags: how-to-guide - Published: 2026-08-14 ### [How to Write and Configure YAML Correlation Rules in SpiderFoot](/smicallef/spiderfoot/how-to-write-and-configure-yaml-correlation-rules-in-spiderfoot) Learn to write and configure YAML correlation rules in SpiderFoot. Automate data analysis and generate findings with this comprehensive guide. - Tags: how-to-guide - Published: 2026-08-14 ### [How to Run SpiderFoot Scans Entirely from the Command Line](/smicallef/spiderfoot/how-to-run-spiderfoot-scans-entirely-from-the-command-line) Master SpiderFoot command line scans for complete OSINT automation. Learn to run local and remote scans efficiently without the web UI using sf.py and sfcli.py. - Tags: how-to-guide - Published: 2026-08-14 ### [How to Configure SpiderFoot Web UI Authentication and Security Settings](/smicallef/spiderfoot/configuring-spiderfoot-web-ui-authentication-and-security-settings) Learn to configure SpiderFoot web UI authentication and security settings. Secure your interface with HTTP Digest Authentication, CSP headers, and TLS encryption. Protect your data today. - Tags: how-to-guide - Published: 2026-08-14 ### [How SpiderFoot's SQLite Database Schema Stores Scan Results and Events](/smicallef/spiderfoot/how-does-spiderfoot-sqlite-database-schema-store-scan-results-and-events) Discover how SpiderFoot's SQLite database schema stores scan results and events in five core tables. Understand event chaining with source_event_hash and tbl_scan_results. - Tags: internals - Published: 2026-08-14 ### [Understanding SpiderFoot's Pub/Sub Event Model and Data Flow Between Modules](/smicallef/spiderfoot/understanding-spiderfoot-pub-sub-event-model-and-data-flow-between-modules) Explore SpiderFoot's pub/sub event model and data flow. Understand how events propagate through modules via thread-safe queues for efficient data discovery. - Tags: internals - Published: 2026-08-14 ### [How SpiderFoot's Module Plugin System Works and How to Create Custom Modules](/smicallef/spiderfoot/how-does-spiderfoot-module-plugin-system-work-and-how-to-create-custom-modules) Explore SpiderFoot's module plugin system. Learn how it works and create your own custom modules with this in-depth guide. - Tags: internals - Published: 2026-08-14 ### [Using Strict Mode (-x Flag) in SpiderFoot Module Activation: Direct Target Consumption](/smicallef/spiderfoot/using-strict-mode-x-flag-in-spiderfoot-module-activation) Unlock SpiderFoot's strict mode with the -x flag for direct target event consumption. Learn how to optimize module execution and minimize data transformations in this guide. - Tags: how-to-guide - Published: 2026-06-23 ### [SpiderFoot Module Execution Priority and Ordering: Controlling Scan Flow with _priority](/smicallef/spiderfoot/spiderfoot-module-execution-priority-and-ordering) Control SpiderFoot scan flow by understanding module execution priority and ordering. Learn how _priority integers dictate the sequence for efficient data gathering. - Tags: internals - Published: 2026-06-23 ### [Tuning Max Thread Configuration for SpiderFoot Performance: A Complete Optimization Guide](/smicallef/spiderfoot/tuning-max-thread-configuration-for-spiderfoot-performance) Optimize SpiderFoot performance by tuning max thread configuration. Learn how to adjust threads for multi-core servers or constrained systems to improve scanning speed. - Tags: performance - Published: 2026-06-23 ### [Overriding DNS Resolution with Custom DNS Servers in SpiderFoot](/smicallef/spiderfoot/overriding-dns-resolution-with-custom-dns-servers-in-spiderfoot) Learn how to override DNS resolution in SpiderFoot using custom DNS servers. Configure the _dnsserver option to direct all queries through your preferred nameserver with dnspython. - Tags: how-to-guide - Published: 2026-06-23 ### [Configuring TOR for Anonymous OSINT Gathering with SpiderFoot: A Complete Guide](/smicallef/spiderfoot/configuring-tor-for-anonymous-osint-gathering-with-spiderfoot) Learn to configure TOR for anonymous OSINT gathering with SpiderFoot. Route traffic via a local TOR SOCKS5 proxy and leverage the sfp_torexits plugin for enhanced privacy and security. - Tags: how-to-guide - Published: 2026-06-23 ### [Configuring SOCKS Proxy for SpiderFoot: Anonymous Reconnaissance Setup Guide](/smicallef/spiderfoot/configuring-socks-proxy-for-spiderfoot) Secure your SpiderFoot reconnaissance by configuring a SOCKS proxy. Learn how to set up anonymous traffic routing for enhanced privacy and data collection with this setup guide. - Tags: how-to-guide - Published: 2026-06-23 ### [Exporting SpiderFoot Scan Results to JSON: A Complete Guide](/smicallef/spiderfoot/exporting-spiderfoot-scan-results-to-json) Learn how to export SpiderFoot scan results to JSON using the built-in HTTP endpoint. Get a complete guide to extracting scan metadata, entities, and module results efficiently. - Tags: how-to-guide - Published: 2026-06-23 ### [Exporting SpiderFoot Scan Results to CSV: Web UI, CLI, and API Methods](/smicallef/spiderfoot/exporting-spiderfoot-scan-results-to-csv) Easily export SpiderFoot scan results to CSV using the Web UI, CLI, or API. Learn how to extract your intelligence data efficiently for analysis and reporting. - Tags: how-to-guide - Published: 2026-06-23 ### [SpiderFoot Command-Line Scanning CLI Reference: Automating OSINT with sfcli.py](/smicallef/spiderfoot/spiderfoot-command-line-scanning-cli-reference) Master SpiderFoot command-line scanning with sfcli.py. Automate OSINT and manage reconnaissance scans programmatically using this interactive CLI reference. - Tags: api-reference - Published: 2026-06-23 ### [SpiderFoot's Pub/Sub Event Model: Technical Architecture Explained](/smicallef/spiderfoot/explanation-of-spiderfoots-pub-sub-event-model) Explore SpiderFoot's pub/sub event model architecture. Understand how modules emit immutable events routed to consumers via a SQLite message bus. Learn more now. - Tags: architecture - Published: 2026-06-23