You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Ahrefs is a large SEO company used by single bloggers to large
enterprises. It may be beneficial to allow (or deny) them in Anubis. They
do publish rDNS entries, so once an Anubis version with TecharoHQ#682 is released,
this policy would benefit from setting up that check.
Crawler information: https://ahrefs.com/robot
Majestic is a UK based specialist search engine and commercial SEO
entity. They claim to "spider the Web for the purpose of building a
search engine" with a distributed crawler. Defaults to allow as it'd be
caught with the generic browser policy definition.
Crawler information: https://mj12bot.com
Screaming Frog is a smaller actor in the SEO space and their crawler
occasionally attempts to access content despite being explicitly
excluded via robots.txt directives. As far as I could research they
neither publish their IP ranges nor provide an information page for
their crawler. That's why this defaults to deny.
Company website: https://www.screamingfrog.co.uk
Checkmark Network is a brand and intellectual property protection
company. If you have no direct business with them, it is likely they
shouldn't be crawling your content in the first place. Defaults to deny
for this reason.
Crawler information: https://www.checkmarknetwork.com/spider.html/
Domainsbot collects information on domains and website data for
intellectual property disputes. Unless you have direct business with
them, there's likely no reason for them to be accessing your content.
Defaults to deny.
Crawler information: https://domainsbot.com/pandalytics/
zoominfo is a data mining and sales platform for enterprise use, feeding
the gathered information into a machine learning model. It is unlikely
to be of value to anyone else. Therefore, this defaults to deny.
Company website: https://www.zoominfo.com
0 commit comments