SpiderFoot

SpiderFoot is an open-source intelligence (OSINT) automation framework written in Python. It's built for security teams, penetration testers, and anyone who needs to map what's publicly known or exposed about a target, whether that's a domain, IP, person, or organization. You run a scan against a target entity, and SpiderFoot's 200+ modules feed data into each other in a publisher/subscriber model, pulling from search engines, threat feeds, breach databases, DNS records, social platforms, and more.

It ships with both a web-based UI and a command-line interface, so it fits into interactive investigation work or automated pipelines. The project has been actively maintained since 2012 and is MIT-licensed.

Key features

• 200+ modules: integrates with sources like Shodan, HaveIBeenPwned, GreyNoise, AlienVault, SecurityTrails, Censys, and dozens of blacklist/threat-intel feeds, most without requiring an API key.
• Dual interface: run scans through the embedded web server or drive everything from the command line.
• Correlation engine: a YAML-configurable rule engine ships with 37 pre-defined rules to surface meaningful findings from raw scan data.
• Broad target support: scan IP addresses, domains/sub-domains, hostnames, CIDR ranges, ASNs, emails, phone numbers, usernames, person names, and Bitcoin addresses.
• Data export: pull results out as CSV, JSON, or GEXF, or query the SQLite backend directly for custom analysis.
• External tool integration: calls out to tools like DNSTwist, Whatweb, Nmap, and CMSeeK to extend recon depth.
• TOR integration: supports dark web searches as part of a scan.
• Docker support: a Dockerfile is provided for containerized deployment.
• Visualisations: built-in graphing of relationships discovered during a scan.

Ideal use cases

SpiderFoot fits red team reconnaissance and penetration test prep, where you need to quickly enumerate subdomains, exposed cloud storage, leaked credentials, and social media footprints for a target. It's also useful defensively: running it against your own domains and infrastructure to see what an attacker would find, catching exposed S3/Azure/DigitalOcean buckets, subdomain hijacking risks, or leaked employee data.

It works well for building a one-off or recurring attack surface snapshot for a specific organization, or for OSINT investigations involving people, emails, or cryptocurrency addresses. The correlation engine helps when you're scanning large targets and need automated flagging of risky findings instead of manually reviewing raw module output.

It's not the right tool if you need real-time, always-on attack surface monitoring with multi-user collaboration, alerting, or a managed API, since those capabilities live in the separate commercial SpiderFoot HX product rather than the open-source version. It also isn't a vulnerability scanner or exploitation framework; it's focused on information gathering, not active exploitation.

Installation

SpiderFoot requires Python 3.7 or later. The maintainers recommend installing a packaged release rather than the git master branch, since master can include untested modules.

Stable build (packaged release):

wget https://github.com/smicallef/spiderfoot/archive/v4.0.tar.gz
tar zxvf v4.0.tar.gz
cd spiderfoot-4.0
pip3 install -r requirements.txt
python3 ./sf.py -l 127.0.0.1:5001

Development build (cloning git master):

git clone https://github.com/smicallef/spiderfoot.git
cd spiderfoot
pip3 install -r requirements.txt
python3 ./sf.py -l 127.0.0.1:5001

Once running, the command above starts the embedded web server on 127.0.0.1:5001, where you access the UI to configure and launch scans. A Dockerfile is also provided for container-based deployment. Documentation and asciinema walkthroughs cover setup of API keys for the tiered/commercial modules and how to write custom correlation rules using the provided template.yaml.