Search Results Utility

A utility for scraping and displaying search results from various sources.

Project Structure

The project consists of two main components:

1. Scraper (/scraper)

A Node.js utility for scraping search results from different sources. It uses Puppeteer for web scraping and can be configured to extract specific data points from search results.

2. Web Interface (/next-web)

A Next.js web application that displays the scraped search results in a modern, responsive interface. Features include:

• Clean, modern UI with dark mode support
• Responsive design for all screen sizes
• Image thumbnails for search results
• Clickable links to original sources
• Search query highlighting

Features

• Web Scraping: Automated data collection from search engines
• Data Processing: Structured data extraction and formatting
• Modern UI: Responsive design with dark mode support
• Performance: Optimized for fast loading and smooth interactions

Getting Started

1. Clone the repository

2. Install dependencies in both directories:

   # Install scraper dependencies
   cd scraper
   npm install
   
   # Install web interface dependencies
   cd ../next-web
   npm install

3. Configure environment variables:

   • Create .env files in both directories as needed
   • See .gitignore for reference on environment file handling

4. Run the development servers:

   # Run scraper
   cd scraper
   npm run dev
   
   # Run web interface
   cd ../next-web
   npm run dev

Development

• The scraper uses Puppeteer for web scraping
• The web interface is built with Next.js and TypeScript
• Styling is done with Tailwind CSS
• Both components use modern ES6+ features

Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a new Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.