This project is a Go (Golang)-based Cyber Threat Intelligence (CTI) collection tool developed as part of the Sibervatan “Yıldız CTI” training program.
Project Objective
It targets hacker-themed .onion sites on the Tor network and collects their HTML content and screenshot data.
Technical Features
- Tor Proxy Integration: Provides an anonymous connection through
127.0.0.1:9050using a SOCKS5 proxy and prevents IP leakage. - Headless Chrome (Chromedp): Renders modern web technologies and captures full-page screenshots.
- Dynamic Directory Structure: Stores each target in a separate directory named after its URL (
example.onion/). - Reporting: Writes scan results to
scan_report.logand the terminal in real time.
Installation and Usage
Requirements
- Go
- Tor Service (must be running in the background)
- Linux / macOS / Windows
Execution Steps
- Clone the Repository:
git clone (https://github.com/Muzoovy4606/CTI-THOR-SCRAPER
cd CTI-THOR-SCRAPER- Install Dependencies:
go mod tidy- Create the targets.yaml File:
For security and privacy reasons, the targets.yaml file is not included in the repository because it contains ransomware and hacker-group links. Before running the project, create a
targets.yamlfile in the project root and add your targets. - Start the Tor Service (Linux/Arch):
sudo systemctl start tor(Note: Make sure the Tor service is listening on port 9050.)
- Run the Tool:
go run main.goUsing the Precompiled Linux Binary
To use the precompiled Linux executable included in the repository without rebuilding the source, run:
chmod +x thor-scraper
./thor-scraper📂 Output Structure
When executed, the program creates the following structure under output/:
output/
├── example.onion/
│ ├── index.html # Sitenin HTML kaynak kodu
│ ├── screenshot.png # Tam sayfa ekran görüntüsü
│ └── site_info.txt # Meta veriler (Başlık, Tarama Tarihi)
└── scan_report.log # Detaylı durum raporu (SUCCESS/FAIL kayıtları)