-
Notifications
You must be signed in to change notification settings - Fork 20
Scan robots.txt #3
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or requesthelp wantedExtra attention is neededExtra attention is needed
Description
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requesthelp wantedExtra attention is neededExtra attention is needed
Make a set of all the endpoints in robots.txt, and add them to the active endpoints.