REK - Reconnaissance Toolkit

A Modern Recon Playbook for Bug Bounty Hunters

REK is a comprehensive reconnaissance toolkit designed for ethical hackers and bug bounty hunters. It provides automated reconnaissance pipelines through sophisticated playbooks, along with modular subdomain enumeration, HTTP status checking, directory scanning, and email search capabilities.

Authors: Jayresearcher, NarutoX, Ninja

🚀 Automated Playbook System

Core Playbook Features

The REK toolkit includes two main automated reconnaissance playbooks:

rek-playbook-v1.sh: Enhanced version with advanced features and better error handling
rek-playbook-v2.sh: Enhanced URL crawler with better outcomes
rek-playbook.sh: Standard reconnaissance pipeline

Quick Start with Playbooks

Interactive Playbook Execution (Recommended)

# Run the interactive menu system
python3 rek.py

# Select option 1: Run Recon Playbook
# Choose your preferred playbook version
# Enter target domain and thread count
# The system automatically installs dependencies and runs the playbook

The interactive mode provides:

Automatic dependency installation via install-script.sh
Playbook version selection
Real-time output streaming
Error handling and validation

Direct Playbook Execution

# Make playbook executable
chmod +x playbook/rek-playbook-v1.sh

# Run basic reconnaissance
./playbook/rek-playbook-v1.sh -d example.com

# Run with custom configuration
./playbook/rek-playbook-v1.sh -d example.com -t 200 --chaos-key YOUR_KEY --github-token YOUR_TOKEN

# Skip specific phases
./playbook/rek-playbook-v1.sh -d example.com --skip-portscan --skip-jsanalysis

Manual Installation

# Install all prerequisites and tools
chmod +x playbook/install-script.sh
./playbook/install-script.sh

📋 Playbook Architecture Wireframe

High-Level System Architecture

┌─────────────────────────────────────────────────────────────────────────────────┐
│                         REK PLAYBOOK SYSTEM ARCHITECTURE                        │
└─────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐    ┌─────────────┐
│   User Input    │    │  Configuration  │    │  Tool Manager   │    │   Results   │
│                 │    │    System       │    │                 │    │  Processor  │
│ • Domain Name   │────│ • API Keys      │────│ • Dependency    │────│ • Markdown  │
│ • CLI Arguments │    │ • Thread Count  │    │   Checking      │    │   Reports   │
│ • Pipeline      │    │ • Skip Flags    │    │ • Installation  │    │ • CSV Files │
│   Settings      │    │ • Tool Paths    │    │ • Version Check │    │ • Cleanup   │
└─────────────────┘    └─────────────────┘    └─────────────────┘    └─────────────┘
         │                       │                       │                       │
         └───────────────────────┼───────────────────────┼───────────────────────┘
                                 │                       │
         ┌───────────────────────▼───────────────────────▼───────────────────────┐
         │                    CORE PLAYBOOK ENGINE                               │
         │                                                                       │
         │  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐      │
         │  │   Phase 1-2     │  │   Phase 3-4     │  │   Phase 5-8     │      │
         │  │   Subdomain     │  │  Live Detection │  │   Content &     │      │
         │  │   Discovery     │  │  & Port Scan    │  │  Vulnerability  │      │
         │  └─────────────────┘  └─────────────────┘  └─────────────────┘      │
         └───────────────────────────────────────────────────────────────────────┘
                                        │
         ┌──────────────────────────────▼──────────────────────────────────────┐
         │                    EXTERNAL TOOL INTEGRATIONS                       │
         │                                                                     │
         │  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐  │
         │  │ Subdomain   │ │ HTTP/Port   │ │ Content     │ │ Analysis    │  │
         │  │ Tools       │ │ Scanners    │ │ Discovery   │ │ Tools       │  │
         │  │             │ │             │ │             │ │             │  │
         │  │ • Subfinder │ │ • HTTPx     │ │ • Gospider  │ │ • GF        │  │
         │  │ • Assetfind │ │ • Naabu     │ │ • Katana    │ │ • Cariddi   │  │
         │  │ • Findomain │ │ • Puredns   │ │ • GAU       │ │ • GetJS     │  │
         │  │ • Chaos     │ │             │ │             │ │             │  │
         │  └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘  │
         └─────────────────────────────────────────────────────────────────────┘

Detailed Playbook Flow Architecture

┌───────────────────────────────────────────────────────────────────────────────────┐
│                           RECONNAISSANCE PIPELINE FLOW                            │
└───────────────────────────────────────────────────────────────────────────────────┘

┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│   PHASE 1   │──▶│   PHASE 2   │──▶│   PHASE 3   │──▶│   PHASE 4   │──▶│   PHASE 5   │
│  Subdomain  │   │  Subdomain  │   │    Live     │   │    Port     │   │  Content    │
│ Enumeration │   │ Permutation │   │ Detection   │   │  Scanning   │   │ Discovery   │
└─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘
       │                 │                 │                 │                 │
       ▼                 ▼                 ▼                 ▼                 ▼
┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│• Subfinder  │   │• DNSGen     │   │• HTTPx      │   │• Naabu      │   │• Gospider   │
│• Assetfinder│   │• Gotator    │   │• Response   │   │• Port List  │   │• Katana     │
│• Findomain  │   │• Goaltdns   │   │  Analysis   │   │• Service    │   │• GAU        │
│• Chaos API  │   │• Permute    │   │• Tech Stack │   │  Detection  │   │• Sitemap    │
│• GitHub API │   │• Puredns    │   │• Headers    │   │• HTTPx Port │   │• Robots.txt │
│• GitLab API │   │• Resolve    │   │• Status     │   │  Probe      │   │• JS Files   │
└─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘

         │                                                                        │
         ▼                                                                        ▼
┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│   PHASE 6   │──▶│   PHASE 7   │──▶│   PHASE 8   │──▶│  REPORTING  │──▶│   CLEANUP   │
│Vulnerability│   │  Endpoint   │   │ JavaScript  │   │  & Summary  │   │& Archival   │
│  Analysis   │   │Categorization│   │  Analysis   │   │  Generation │   │             │
└─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘
       │                 │                 │                 │                 │
       ▼                 ▼                 ▼                 ▼                 ▼
┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│• GF Patterns│   │• File Types │   │• GetJS      │   │• Markdown   │   │• Archive    │
│• XSS Filter │   │• JSON Files │   │• Secret     │   │  Report     │   │• Temp File  │
│• SQLi Filter│   │• Config     │   │  Detection  │   │• Statistics │   │  Removal    │
│• SSRF Filter│   │• Backup     │   │• Cariddi    │   │• Findings   │   │• Results    │
│• LFI Filter │   │• PDF Files  │   │• API Keys   │   │• Next Steps │   │  Structure  │
│• RCE Filter │   │• Log Files  │   │• Tokens     │   │• Timeline   │   │• Validation │
└─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘

Tool Integration Matrix

┌─────────────────────────────────────────────────────────────────────────────────┐
│                          TOOL INTEGRATION & DATA FLOW                           │
└─────────────────────────────────────────────────────────────────────────────────┘

    INPUT                 PROCESSING                      OUTPUT
┌─────────────┐     ┌─────────────────────────┐     ┌─────────────────────┐
│   Domain    │────▶│    Subdomain Tools      │────▶│  Raw Subdomains     │
│ example.com │     │                         │     │                     │
└─────────────┘     │ ┌─────────────────────┐ │     │ • subfinder.txt     │
                    │ │   Subfinder         │ │     │ • assetfinder.txt   │
                    │ │   • DNS Brute       │ │     │ • findomain.txt     │
                    │ │   • CT Logs         │ │     │ • chaos.txt         │
                    │ │   • Certificate     │ │     │ • github-subs.txt   │
                    │ │     Transparency    │ │     │ • gitlab-subs.txt   │
                    │ └─────────────────────┘ │     └─────────────────────┘
                    │                         │               │
                    │ ┌─────────────────────┐ │               ▼
                    │ │   Assetfinder       │ │     ┌─────────────────────┐
                    │ │   • API Sources     │ │     │   Deduplicated      │
                    │ │   • Search Engines  │ │────▶│   Subdomain List    │
                    │ └─────────────────────┘ │     │                     │
                    │                         │     │ • sorted-subs.txt   │
                    │ ┌─────────────────────┐ │     │ • 1000+ subdomains  │
                    │ │   External APIs     │ │     └─────────────────────┘
                    │ │   • Chaos Project   │ │               │
                    │ │   • GitHub Commits  │ │               ▼
                    │ │   • GitLab Repos    │ │     ┌─────────────────────┐
                    │ └─────────────────────┘ │     │    Permutation      │
                    └─────────────────────────┘     │     Generation      │
                                                    │                     │
                                                    │ • DNSGen            │
                                                    │ • Gotator           │
                                                    │ • Goaltdns          │
                                                    │ • Custom Wordlists  │
                                                    └─────────────────────┘
                                                              │
                                                              ▼
                                                    ┌─────────────────────┐
                                                    │   DNS Resolution    │
                                                    │                     │
                                                    │ • Puredns           │
                                                    │ • Mass DNS          │
                                                    │ • Custom Resolvers  │
                                                    │ • Validation        │
                                                    └─────────────────────┘
                                                              │
                                                              ▼
                                                    ┌─────────────────────┐
                                                    │   Live Detection    │
                                                    │                     │
                                                    │ • HTTPx Probing     │
                                                    │ • Status Codes      │
                                                    │ • Technology Stack  │
                                                    │ • Response Headers  │
                                                    └─────────────────────┘

Configuration and Pipeline Management

┌─────────────────────────────────────────────────────────────────────────────────┐
│                        CONFIGURATION & PIPELINE SYSTEM                          │
└─────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Configuration  │    │   Pipeline      │    │    Results      │
│    Manager      │    │   Executor      │    │   Management    │
│                 │    │                 │    │                 │
│ ┌─────────────┐ │    │ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │ config.conf │ │    │ │ Phase Logic │ │    │ │ Timestamped │ │
│ │             │ │    │ │             │ │    │ │ Directories │ │
│ │ • API Keys  │ │◄──►│ │ • Execution │ │◄──►│ │             │ │
│ │ • Threads   │ │    │ │   Control   │ │    │ │ • Subdomains│ │
│ │ • Timeouts  │ │    │ │ • Skip Logic│ │    │ │ • Endpoints │ │
│ │ • Paths     │ │    │ │ • Error     │ │    │ │ • JS Files  │ │
│ └─────────────┘ │    │ │   Handling  │ │    │ │ • Vulns     │ │
│                 │    │ └─────────────┘ │    │ └─────────────┘ │
│ ┌─────────────┐ │    │                 │    │                 │
│ │ Pipeline    │ │    │ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │ Storage     │ │    │ │ Progress    │ │    │ │   Report    │ │
│ │             │ │    │ │ Tracking    │ │    │ │ Generation  │ │
│ │ • Saved     │ │    │ │             │ │    │ │             │ │
│ │   Settings  │ │    │ │ • Spinner   │ │    │ │ • Markdown  │ │
│ │ • Domain    │ │    │ │ • Timers    │ │    │ │ • Summary   │ │
│ │   History   │ │    │ │ • Logging   │ │    │ │ • Statistics│ │
│ └─────────────┘ │    │ └─────────────┘ │    │ └─────────────┘ │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Tool Installation Architecture

┌─────────────────────────────────────────────────────────────────────────────────┐
│                        AUTOMATED INSTALLATION SYSTEM                            │
└─────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│    System       │    │   Language      │    │     Tool        │
│   Detection     │    │   Runtimes      │    │  Installation   │
│                 │    │                 │    │                 │
│ • OS Type       │    │ • Go Lang       │    │ • GitHub Repos  │
│ • Architecture  │────│ • Python 3      │────│ • Binary Downloads│
│ • Package Mgr   │    │ • Node.js       │    │ • Compilation   │
│ • Permissions   │    │ • Dependencies  │    │ • Path Setup    │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
         ┌───────────────────────▼───────────────────────┐
         │              VERIFICATION SYSTEM              │
         │                                               │
         │  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
         │  │Tool Version │ │ Dependency  │ │ Integration │
         │  │  Checking   │ │  Validation │ │   Testing   │
         │  │             │ │             │ │             │
         │  │ • Command   │ │ • Libraries │ │ • Tool      │
         │  │   Available │ │ • Paths     │ │   Execution │
         │  │ • Version   │ │ • Resolvers │ │ • Output    │
         │  │   Compare   │ │ • Wordlists │ │   Parsing   │
         │  └─────────────┘ └─────────────┘ └─────────────┘
         └───────────────────────────────────────────────────┘

🛠️ Playbook Command Reference

Basic Usage

# Quick reconnaissance
./playbook/rek-playbook-v1.sh -d target.com

# Custom thread count
./playbook/rek-playbook-v1.sh -d target.com -t 200

# Use API keys for enhanced results
./playbook/rek-playbook-v1.sh -d target.com \
  --chaos-key YOUR_CHAOS_KEY \
  --github-token YOUR_GITHUB_TOKEN \
  --gitlab-token YOUR_GITLAB_TOKEN

Phase Control

# Skip specific phases
./playbook/rek-playbook-v1.sh -d target.com \
  --skip-portscan \
  --skip-jsanalysis \
  --skip-vulnanalysis

# Run only subdomain discovery
./playbook/rek-playbook-v1.sh -d target.com \
  --skip-permutation \
  --skip-fingerprint \
  --skip-portscan \
  --skip-spider \
  --skip-vulnanalysis \
  --skip-jsanalysis

Configuration Options

# Custom configuration file
./playbook/rek-playbook-v1.sh -d target.com -c /path/to/config.conf

# Custom output directory
./playbook/rek-playbook-v1.sh -d target.com -o /path/to/results

# Help and options
./playbook/rek-playbook-v1.sh --help

📊 Output Structure

Directory Layout

results/
└── target.com-20240101-120000/
    ├── subdomains/
    │   ├── sorted-subs.txt
    │   ├── subs-alive.txt
    │   ├── httpx-output.txt
    │   └── subs-portscanned.txt
    ├── endpoints/
    │   ├── spider-output.txt
    │   ├── json-endpoints.txt
    │   └── backup-endpoints.txt
    ├── js/
    │   ├── getjs-output.txt
    │   └── js-secrets.txt
    ├── vulnerabilities/
    │   ├── checkfor-xss.txt
    │   ├── checkfor-sqli.txt
    │   └── checkfor-ssrf.txt
    └── recon-report.md

Generated Report

Markdown Summary: Complete reconnaissance report with statistics
CSV Exports: Machine-readable data for further analysis
Categorized Findings: Organized by vulnerability type and file extension
Next Steps: Actionable recommendations for manual testing

🔧 Additional REK Features

Core Modules

Subdomain Enumeration: Multi-source subdomain discovery using DNS Dumpster, Certificate Transparency, and DNS brute-forcing
HTTP Status Checking: Concurrent HTTP/HTTPS probing with detailed response analysis
Directory Scanning: Technology-aware directory and file discovery with screenshot capabilities
Email Search: GitHub-based email harvesting with breach detection via HIBP

Advanced Capabilities

Technology Detection: Automatic web technology identification for targeted scanning
Parallel Processing: Async/await implementation for maximum performance
Screenshot Capture: Automated visual documentation of discovered endpoints
Breach Intelligence: Integration with Have I Been Pwned API
Custom Wordlists: Domain-specific wordlist generation and global wordlist learning

🚀 Installation

Prerequisites

# Install Python dependencies
pip install -r requirements.txt

# Install Chrome/Chromium for screenshot functionality
# Ubuntu/Debian:
sudo apt-get install chromium-browser

# macOS:
brew install chromium

Tool Dependencies

The toolkit includes installation scripts for external tools:

# Run the installation script for your platform
chmod +x playbook/install-script.sh
./playbook/install-script.sh

📚 Technical Usage

Command Line Interface

1. Subdomain Enumeration

# Basic subdomain enumeration
python3 rek.py -d example.com -o results.txt

# Advanced enumeration with custom wordlist and GitHub token
python3 rek.py -d example.com \
  -w wordlists/subdomains-top5000.txt \
  --token ghp_your_github_token \
  -t 15 -c 100 --limit-commits 50

2. HTTP Status Checking

# Check HTTP status for discovered subdomains
python3 rek.py --input results.txt -o http_results.csv -t 10 -c 50

3. Directory Scanning

# Scan live subdomains for directories and files
python3 rek.py --input http_results.csv --status 200,301,403 \
  --dir-wordlist wordlists/common-paths.txt \
  --depth 5 -t 10 -c 30

4. Email Search

# Search by domain
python3 rek.py --email-domain example.com \
  --token ghp_your_github_token \
  --hibp-key your_hibp_api_key \
  -o email_results.csv

# Search by organization/username
python3 rek.py --org microsoft \
  --token ghp_your_github_token \
  --limit-commits 100

Interactive Mode

Main Menu Options

python3 rek.py

# Main Menu Options:
# 1. Run Recon Playbook    - Execute automated reconnaissance playbooks
# 2. Subdomain Enumeration - Discover subdomains using multiple techniques  
# 3. HTTP Status Checking  - Check HTTP status of discovered domains
# 4. Directory Scanning    - Scan for directories and files on web servers
# 5. REK Email Search      - Search for email addresses in GitHub repositories
# 6. REK Wordlist Generator- Generate and download wordlists for testing
# 7. Exit                  - Exit the application

Command Line Help

# Get detailed help information
python3 rek.py --help

# Or use the short form
python3 rek.py -h

Detailed Parameter Reference

Subdomain Enumeration Parameters

python3 rek.py -d example.com [OPTIONS]

Required:
  -d, --domain DOMAIN         Target domain (e.g., example.com)

Optional:
  -w, --subdomain-wordlist    Custom wordlist for subdomain enumeration
  -o, --output FILE          Output file (default: results.txt)
  --token TOKEN              GitHub Personal Access Token for enhanced results
  --limit-commits N          Max commits to scan per repo (default: 50)
  --skip-forks              Skip forked repositories during GitHub search
  -t, --timeout N           Request timeout in seconds (default: 10)
  -c, --concurrency N       Maximum concurrent requests (default: 50)
  -r, --retries N           Number of retries for failed requests (default: 3)
  --silent                  Run in silent mode (minimal output)

Example:
python3 rek.py -d example.com -w wordlists/subdomains.txt --token ghp_xxx -t 15 -c 100

HTTP Status Checking Parameters

python3 rek.py --input FILE [OPTIONS]

Required:
  --input FILE              Input file with URLs to check

Optional:
  -o, --output FILE         Output CSV file (default: http_results.csv)
  -t, --timeout N           Request timeout in seconds (default: 10)
  -c, --concurrency N       Maximum concurrent requests (default: 50)
  --silent                  Run in silent mode (minimal output)

Example:
python3 rek.py --input results.txt -o http_results.csv -t 15 -c 100

Directory Scanning Parameters

python3 rek.py --input FILE --status CODES [OPTIONS]
# OR
python3 rek.py --url URL [OPTIONS]

Required (Option 1):
  --input FILE              Input CSV file with URLs and status codes
  --status CODES            Comma-separated status codes (e.g., 200,301,403)

Required (Option 2):
  --url URL                 Single URL to scan directly

Optional:
  --dir-wordlist FILE       Custom wordlist for directory scanning
  --depth N                 Maximum crawling depth (1-10, default: 5)
  -t, --timeout N           Request timeout in seconds (default: 10)
  -c, --concurrency N       Maximum concurrent requests (default: 50)
  --silent                  Run in silent mode (minimal output)

Examples:
python3 rek.py --input http_results.csv --status 200,301,403 --depth 3
python3 rek.py --url https://example.com --dir-wordlist wordlists/common.txt

Email Search Parameters

# Search by domain
python3 rek.py --email-domain DOMAIN [OPTIONS]

# Search by GitHub username
python3 rek.py --email-username USERNAME [OPTIONS]

# Search by GitHub organization
python3 rek.py --org ORGANIZATION [OPTIONS]

Required (choose one):
  --email-domain DOMAIN     Domain for email search
  --email-username USER     GitHub username for email search
  --org ORGANIZATION        GitHub organization for email search

Optional:
  --token TOKEN             GitHub Personal Access Token (recommended)
  --hibp-key KEY            Have I Been Pwned API key for breach checking
  --limit-commits N         Max commits to scan per repo (default: 50)
  --skip-forks              Skip forked repositories during search
  -o, --output FILE         Output CSV file (default: email_results.csv)
  -t, --timeout N           Request timeout in seconds (default: 10)
  --silent                  Run in silent mode (minimal output)

Examples:
python3 rek.py --email-domain example.com --token ghp_xxx --hibp-key xxx
python3 rek.py --org microsoft --token ghp_xxx --limit-commits 100
python3 rek.py --email-username johndoe --token ghp_xxx --skip-forks

REK Wordlist Generator

The wordlist generator is available through the interactive menu (option 6) and provides:

Features:

Download SecLists wordlists by category
Generate domain-specific custom wordlists
Merge multiple wordlists with deduplication
List and manage existing wordlists
Clean up old or duplicate wordlists

Categories Available:

Subdomains (basic and advanced)
Directories (basic and advanced)
Files (basic and advanced)
Parameters (basic and advanced)
Vulnerabilities (XSS, SQLi, LFI, RCE)
API endpoints and methods

Technology-Specific Wordlists:

WordPress, Drupal, Joomla
Laravel, Django, Node.js
Apache, Nginx, IIS
PHP, Python, Java

Usage:

# Access through interactive menu
python3 rek.py
# Select option 6: REK Wordlist Generator

# Or run the standalone generator
python3 advanced_wordlist_generator.py

🤝 Contributing

Fork the repository
Create a feature branch
Make your changes
Test thoroughly
Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

⚠️ Disclaimer

This tool is for educational and authorized security testing purposes only. Users are responsible for ensuring they have proper authorization before testing any systems.

Name		Name	Last commit message	Last commit date
Latest commit History 19 Commits
attached_assets		attached_assets
playbook		playbook
wordlists		wordlists
.gitignore		.gitignore
.replit		.replit
LICENSE.txt		LICENSE.txt
README.md		README.md
__init__.py		__init__.py
config.conf		config.conf
rek-beta.py		rek-beta.py
rek.py		rek.py
rek_email_search.py		rek_email_search.py
rek_wordlist_generator.py		rek_wordlist_generator.py
requirements.txt		requirements.txt
resolvers-trusted.txt		resolvers-trusted.txt
resolvers.txt		resolvers.txt

License

workflow-builder/rek

Folders and files

Latest commit

History

Repository files navigation