Robotex

An automated, open-source bot protection tool for web developers.

Robotex automatically pulls from community-maintained blocklists (like ai-robots-txt/ai.robots.txt) and allows you to merge them with your custom rules. It serves two purposes:

1. A CLI tool to automatically generate a standard robots.txt file for static sites.
2. An active middleware function to actively block bad bots on server-side frameworks (Next.js, Node, etc.).

Installation

Install locally in your project: npm install robotex

1. CLI Usage (For Static Sites like GitHub Pages)

You can run the generator manually: npx robotex

Or add it to your package.json build script to run automatically during deployment: "scripts": { "build": "robotex && your-build-command" }

CLI Commands

• npx robotex - Generates the robots.txt file.
• npx robotex init - Generates a starter robotex.json config file for custom rules.
• npx robotex --dry-run - Previews your robots.txt in the terminal without saving it.

Note: Robotex will also automatically search your public folders for a sitemap.xml and include it if found!

Configuration (Optional)

If you want to add custom rules, create a robotex.json file in the root of your project:

{
"sitemap": "https://mysite.com/sitemap.xml",
"allowBots": ["Googlebot"],
"blockBots": ["MyCustomScraper"],
"output": "public/robots.txt"
}

2. Server Usage (For Frameworks like Next.js or Node)

Malicious bots often ignore robots.txt files. To actively block them, use the exported isBot function in your server's middleware.

The function automatically fetches and caches the community blocklist for 24 hours.

Example using Next.js Middleware:

import { isBot } from 'robotex';
import { NextResponse } from 'next/server';

export async function middleware(request) {
    const userAgent = request.headers.get('user-agent');

    if (await isBot(userAgent)) {
        return new NextResponse('Access Denied', { status: 403 });
    }

    return NextResponse.next();
}