NodeJS robots.txt parser with support for wildcard (*) matching.
-
Updated
May 12, 2026 - JavaScript
#
rfc-9309
Here are 4 public repositories matching this topic...
A parser for robots.txt with support for wildcards. See also RFC 9309.
parser erlang parsing crawling robots-txt parsing-library robots-parser erlang-library robots-exclusion-standard rfc-9309
-
Updated
Nov 21, 2023 - Erlang
Parse robots.txt and sitemaps using dotnet
-
Updated
Mar 6, 2026 - C#
Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
python geo robots-txt rfc-9309 gptbot generative-engine-optimization perplexitybot claudebot crawler-policy compliance-geo llm-crawlers
-
Updated
May 10, 2026 - Python
Improve this page
Add a description, image, and links to the rfc-9309 topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the rfc-9309 topic, visit your repo's landing page and select "manage topics."