Katana

Katana is a command-line web crawler built for security testing and reconnaissance pipelines. It's aimed at penetration testers, bug bounty hunters, and anyone automating URL discovery against a target site. The project is written in Go and maintained by ProjectDiscovery, the team behind other recon tools like nuclei and httpx.

The crawler supports two execution modes: a standard HTTP-based crawler and a headless browser mode that executes JavaScript, which matters when crawling single-page apps or sites that render links dynamically. It ships with scope controls, content deduplication, and output formatting designed to plug into larger automation workflows rather than run as a one-off manual tool.

Key features

• Dual crawling modes: standard (fast, HTTP-only) and headless (via Chrome, for JS-heavy sites)
• JavaScript parsing: extracts endpoints from JS files, with an optional jsluice mode for deeper (but more memory-intensive) parsing
• Automatic form filling: experimental support for filling and submitting forms during a crawl
• Scope control: restrict crawling by regex, root domain, fully qualified domain, or custom field
• Knowledge base classification: ML-based page-type and form classification, plus optional secret and endpoint (REST/GraphQL/SOAP/XHR) extraction
• Content similarity filtering: dedupe near-identical pages using simhash, TF-IDF, or BM25, on top of exact MD5 content dedup
• Flexible input/output: accepts URLs, lists, or stdin; outputs to stdout, file, or JSONL with customizable field selection
• Rate limiting and concurrency controls: per-second/per-minute limits, per-host limits, and configurable concurrency/parallelism
• Resume support: pause and resume scans using a resume config file
• TLS impersonation and proxy support: experimental JA3 randomization and HTTP/SOCKS5 proxy routing

Ideal use cases

Katana fits well into recon and attack-surface mapping pipelines where you need a list of URLs, endpoints, or JS-derived paths from a target domain. It's commonly chained with tools like httpx for input and nuclei for follow-up scanning, since its JSONL output and field selection are built for piping into other CLI tools.

It's also useful for crawling JavaScript-heavy applications where a standard crawler would miss client-side rendered links, thanks to the headless mode with configurable page-load strategies.

It's not the right tool if you need a general-purpose, polite web crawler for content indexing or archival (there's no robots.txt-respecting default behavior beyond the optional known-files crawling, and the tool's warning banner makes clear it's meant for authorized security testing). It's also not suited for large-scale distributed crawling across thousands of domains in parallel without building orchestration around it yourself, since it's a single-binary CLI tool, not a crawling service.

Installation

Katana requires Go to build from source. Install the binary directly:

CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest

Alternatively, download a pre-compiled binary from the GitHub releases page.

Docker:

docker pull projectdiscovery/katana:latest
docker run projectdiscovery/katana:latest -u https://tesla.com

For headless mode in Docker:

docker run projectdiscovery/katana:latest -u https://tesla.com -system-chrome -headless

Ubuntu, install prerequisites including Go and Chrome, then install katana the same way:

sudo apt update
sudo apt install zip curl wget git snapd
sudo snap refresh
sudo snap install golang --classic
go install github.com/projectdiscovery/katana/cmd/katana@latest

Once installed, run katana -h to see all available flags, or start a basic crawl with:

katana -u https://example.com