CTI Web Scraper

A Go-based tool for automated reconnaissance and information gathering.

This project uses Golang and Chromedp to collect HTML, screenshots, and a link map from target websites without requiring manual interaction.

Core Features

• Headless Browser Automation: Processes modern, JavaScript-heavy websites in the background.
• Screenshot Capture: Saves the current state of the target website as a high-resolution .png file.
• Source Code Analysis: Inspects the site’s DOM and downloads its HTML source.
• Link Mapping: Parses and lists all external and internal links (href) on the page.
• Status Code Handling: Detects network errors and conditions such as 404 (Page Not Found), avoiding unnecessary collection attempts.
• Automatic File Organization: Sanitizes invalid URL characters and stores all output in an organized structure under outputs/.

Installation

Requirements

• Go (Golang) must be installed on your system.
• Google Chrome or Chromium must be installed.

Clone the Project

git clone https://github.com/muzoovy/Tools.git
cd Tools/Cti-Web-Scraper

Install Dependencies

go mod tidy

Usage

Run the tool by specifying the target website with the -url parameter.

go run cti-web-scraper.go -url [https://hedef-site.com]

Build (Optional)

To compile the tool into a portable .exe file:

go build -o scraper
./scraper -url [https://hedef-site.com]

📂 Output Structure

When executed, the program creates an outputs/ directory in the project root and stores all collected data there.

cti-web-scraper/
├── cti-web-scraper.go
├── outputs/
│   ├── html_hedef_site_com.html      # Kaynak Kod Dosyası
│   ├── ss_hedef_site_com.png         # Ekran Görüntüsü
│   └── linkler_hedef_site_com.txt    # Link Listesi
└── README.md