feat(config): Add FCrDNS checker #682
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hey, thanks for the contribution! The CHALLENGE rule is more meant for client-facing challenges. You probably want something like the checker.Impl interface here. This will let you add an |
Axelen123
changed the title
Axelen123
commented
Jun 22, 2025
|
I have finished converting the implementation to a checker. |
Xe
approved these changes
Jun 25, 2025
|
I have added CEL bindings and reverted to the previous checker behavior. |
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
If a client claims to be Googlebot but isn't from Google, that's kinda suspicious and should be treated as such. Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
Xe
approved these changes
Jun 27, 2025
|
Thanks much! This is gonna let us do a lot of fun things :) |
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
herrbischoff
added a commit
to herrbischoff/anubis
that referenced
this pull request
Jul 16, 2025
Ahrefs is a large SEO company used by single bloggers to large enterprises. It may be beneficial to allow (or deny) them in Anubis. They do publish rDNS entries, so once an Anubis version with TecharoHQ#682 is released, this policy would benefit from setting up that check. Further information: https://ahrefs.com/robot
herrbischoff
added a commit
to herrbischoff/anubis
that referenced
this pull request
Jul 16, 2025
Ahrefs is a large SEO company used by single bloggers to large enterprises. It may be beneficial to allow (or deny) them in Anubis. They do publish rDNS entries, so once an Anubis version with TecharoHQ#682 is released, this policy would benefit from setting up that check. Crawler information: https://ahrefs.com/robot Majestic is a UK based specialist search engine and commercial SEO entity. They claim to "spider the Web for the purpose of building a search engine" with a distributed crawler. Defaults to allow as it'd be caught with the generic browser policy definition. Crawler information: https://mj12bot.com Screaming Frog is a smaller actor in the SEO space and their crawler occasionally attempts to access content despite being explicitly excluded via robots.txt directives. As far as I could research they neither publish their IP ranges nor provide an information page for their crawler. That's why this defaults to deny. Company website: https://www.screamingfrog.co.uk Checkmark Network is a brand and intellectual property protection company. If you have no direct business with them, it is likely they shouldn't be crawling your content in the first place. Defaults to deny for this reason. Crawler information: https://www.checkmarknetwork.com/spider.html/ Domainsbot collects information on domains and website data for intellectual property disputes. Unless you have direct business with them, there's likely no reason for them to be accessing your content. Defaults to deny. Crawler information: https://domainsbot.com/pandalytics/ zoominfo is a data mining and sales platform for enterprise use, feeding the gathered information into a machine learning model. It is unlikely to be of value to anyone else. Therefore, this defaults to deny. Company website: https://www.zoominfo.com
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #431. This PR implements dynamic verification of bot IPs using DNS records. For details regarding how it works, see the documentation I have added. I am not sure if the way I added a new
algorithmis the best way to implement this. Let me know if there is a better way.[Unreleased]section of docs/docs/CHANGELOG.mdnpm run test:integration(unsupported on Windows, please use WSL)