Skip to content

feat(config): Add FCrDNS checker #682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

feat(config): Add FCrDNS checker #682

Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
cf59306
implement fcrdns challenge
Axelen123 Jun 16, 2025
76c85a5
Merge remote-tracking branch 'origin/main'
Axelen123 Jun 16, 2025
6d71ea2
use algorithm instead
Axelen123 Jun 16, 2025
1d3ed05
add documentation
Axelen123 Jun 17, 2025
1b0a36b
Merge remote-tracking branch 'origin/main' into fcrdns
Axelen123 Jun 17, 2025
c6869b3
implement as a checker instead
Axelen123 Jun 22, 2025
4cc4c33
Merge remote-tracking branch 'origin/main' into fcrdns
Axelen123 Jun 22, 2025
17cfb24
cleanup
Axelen123 Jun 22, 2025
ede9476
Merge remote-tracking branch 'origin/main' into fcrdns
Axelen123 Jun 25, 2025
5fbb70f
update documentation
Axelen123 Jun 25, 2025
fee4bd9
add fcrdns function to cel
Axelen123 Jun 26, 2025
c44a030
revert multiple checker behavior
Axelen123 Jun 26, 2025
8018a5b
fully revert multi checker behavior
Axelen123 Jun 26, 2025
98c69c9
Merge remote-tracking branch 'origin/main' into fcrdns
Axelen123 Jun 26, 2025
5feae37
fix formatting
Axelen123 Jun 26, 2025
be7900b
chore(lib/policy): run go tool goimports
Xe Jun 27, 2025
c726381
docs(CHANGELOG): update changelog notes
Xe Jun 27, 2025
c5bb34b
feat(data/crawlers): add weight to clients that lie
Xe Jun 27, 2025
d1254e0
Merge branch 'main' into fcrdns
Xe Jun 27, 2025
01b8b87
chore: spelling
Xe Jun 27, 2025
dfac612
chore(data/crawlers/googlebot): fix typo
Xe Jun 27, 2025
64989c4
chore: fix spelling errors
Xe Jun 27, 2025
ff978d2
tests(lib): inject fcrdns into the test contexts
Xe Jun 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion cmd/anubis/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
"github.com/TecharoHQ/anubis"
"github.com/TecharoHQ/anubis/data"
"github.com/TecharoHQ/anubis/internal"
"github.com/TecharoHQ/anubis/internal/fcrdns"

Check failure on line 33 in cmd/anubis/main.go

View workflow job for this annotation

GitHub Actions / Check Spelling 

`fcrdns` is not a recognized word. (unrecognized-spelling)
"github.com/TecharoHQ/anubis/internal/thoth"
libanubis "github.com/TecharoHQ/anubis/lib"
botPolicy "github.com/TecharoHQ/anubis/lib/policy"
Expand Down Expand Up @@ -278,7 +279,8 @@
log.Fatalf("you can't set COOKIE_DOMAIN and COOKIE_DYNAMIC_DOMAIN at the same time")
}

ctx := context.Background()
fdns := fcrdns.NewFCrDNS()

Check failure on line 282 in cmd/anubis/main.go

View workflow job for this annotation

GitHub Actions / Check Spelling 

`FCr` is not a recognized word. (unrecognized-spelling)

Check failure on line 282 in cmd/anubis/main.go

View workflow job for this annotation

GitHub Actions / Check Spelling 

`fdns` is not a recognized word. (unrecognized-spelling)

Check warning on line 282 in cmd/anubis/main.go

View workflow job for this annotation

GitHub Actions / Check Spelling 

`fcrdns` is not a recognized word. (unrecognized-spelling)
ctx := fcrdns.With(context.Background(), fdns)

Check warning on line 283 in cmd/anubis/main.go

View workflow job for this annotation

GitHub Actions / Check Spelling 

`fdns` is not a recognized word. (unrecognized-spelling)

Check warning on line 283 in cmd/anubis/main.go

View workflow job for this annotation

GitHub Actions / Check Spelling 

`fcrdns` is not a recognized word. (unrecognized-spelling)

// Thoth configuration
switch {
Expand Down Expand Up @@ -384,6 +386,7 @@
}

s, err := libanubis.New(libanubis.Options{
FCrDNS: fdns,

Check warning on line 389 in cmd/anubis/main.go

View workflow job for this annotation

GitHub Actions / Check Spelling 

`fdns` is not a recognized word. (unrecognized-spelling)

Check warning on line 389 in cmd/anubis/main.go

View workflow job for this annotation

GitHub Actions / Check Spelling 

`FCr` is not a recognized word. (unrecognized-spelling)
BasePrefix: *basePrefix,
StripBasePrefix: *stripBasePrefix,
Next: rp,
Expand Down
28 changes: 12 additions & 16 deletions data/crawlers/applebot.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,16 @@
# Indexing for search and Siri
# https://support.apple.com/en-us/119829
- name: applebot
user_agent_regex: Applebot
expression:
all:
- userAgent.matches("Applebot")
- fcrdns.check("\\.applebot\\.apple\\.com$")
action: ALLOW
# https://search.developer.apple.com/applebot.json
remote_addresses: [
"17.241.208.160/27",
"17.241.193.160/27",
"17.241.200.160/27",
"17.22.237.0/24",
"17.22.245.0/24",
"17.22.253.0/24",
"17.241.75.0/24",
"17.241.219.0/24",
"17.241.227.0/24",
"17.246.15.0/24",
"17.246.19.0/24",
"17.246.23.0/24",
]
- name: not-applebot
expression:
all:
- userAgent.matches("Applebot")
- '!(fcrdns.check("\\.applebot\\.apple\\.com$"))'
action: WEIGH
weight:
adjust: 5
45 changes: 13 additions & 32 deletions data/crawlers/bingbot.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,34 +1,15 @@
- name: bingbot
user_agent_regex: \+http\://www\.bing\.com/bingbot\.htm
# https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26
expression:
all:
- userAgent.matches("\\+http\\://www\\.bing\\.com/bingbot\\.htm")
- fcrdns.check("\\.search\\.msn\\.com$")
action: ALLOW
# https://www.bing.com/toolbox/bingbot.json
remote_addresses: [
"157.55.39.0/24",
"207.46.13.0/24",
"40.77.167.0/24",
"13.66.139.0/24",
"13.66.144.0/24",
"52.167.144.0/24",
"13.67.10.16/28",
"13.69.66.240/28",
"13.71.172.224/28",
"139.217.52.0/28",
"191.233.204.224/28",
"20.36.108.32/28",
"20.43.120.16/28",
"40.79.131.208/28",
"40.79.186.176/28",
"52.231.148.0/28",
"20.79.107.240/28",
"51.105.67.0/28",
"20.125.163.80/28",
"40.77.188.0/22",
"65.55.210.0/24",
"199.30.24.0/23",
"40.77.202.0/24",
"40.77.139.0/25",
"20.74.197.0/28",
"20.15.133.160/27",
"40.77.177.0/24",
"40.77.178.0/23"
]
- name: not-bingbot
expression:
all:
- userAgent.matches("\\+http\\://www\\.bing\\.com/bingbot\\.htm")
- '!(fcrdns.check("\\.search\\.msn\\.com$"))'
action: WEIGH
weight:
adjust: 5
274 changes: 13 additions & 261 deletions data/crawlers/googlebot.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,263 +1,15 @@
- name: googlebot
user_agent_regex: \+http\://www\.google\.com/bot\.html
# https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
expression:
all:
- userAgent.matches("\\+http\\://www\\.google\\.com/bot\\.html")
- fcrdns.check("\\.googlebot\\.com$")
action: ALLOW
# https://developers.google.com/static/search/apis/ipranges/googlebot.json
remote_addresses: [
"2001:4860:4801:10::/64",
"2001:4860:4801:11::/64",
"2001:4860:4801:12::/64",
"2001:4860:4801:13::/64",
"2001:4860:4801:14::/64",
"2001:4860:4801:15::/64",
"2001:4860:4801:16::/64",
"2001:4860:4801:17::/64",
"2001:4860:4801:18::/64",
"2001:4860:4801:19::/64",
"2001:4860:4801:1a::/64",
"2001:4860:4801:1b::/64",
"2001:4860:4801:1c::/64",
"2001:4860:4801:1d::/64",
"2001:4860:4801:1e::/64",
"2001:4860:4801:1f::/64",
"2001:4860:4801:20::/64",
"2001:4860:4801:21::/64",
"2001:4860:4801:22::/64",
"2001:4860:4801:23::/64",
"2001:4860:4801:24::/64",
"2001:4860:4801:25::/64",
"2001:4860:4801:26::/64",
"2001:4860:4801:27::/64",
"2001:4860:4801:28::/64",
"2001:4860:4801:29::/64",
"2001:4860:4801:2::/64",
"2001:4860:4801:2a::/64",
"2001:4860:4801:2b::/64",
"2001:4860:4801:2c::/64",
"2001:4860:4801:2d::/64",
"2001:4860:4801:2e::/64",
"2001:4860:4801:2f::/64",
"2001:4860:4801:31::/64",
"2001:4860:4801:32::/64",
"2001:4860:4801:33::/64",
"2001:4860:4801:34::/64",
"2001:4860:4801:35::/64",
"2001:4860:4801:36::/64",
"2001:4860:4801:37::/64",
"2001:4860:4801:38::/64",
"2001:4860:4801:39::/64",
"2001:4860:4801:3a::/64",
"2001:4860:4801:3b::/64",
"2001:4860:4801:3c::/64",
"2001:4860:4801:3d::/64",
"2001:4860:4801:3e::/64",
"2001:4860:4801:40::/64",
"2001:4860:4801:41::/64",
"2001:4860:4801:42::/64",
"2001:4860:4801:43::/64",
"2001:4860:4801:44::/64",
"2001:4860:4801:45::/64",
"2001:4860:4801:46::/64",
"2001:4860:4801:47::/64",
"2001:4860:4801:48::/64",
"2001:4860:4801:49::/64",
"2001:4860:4801:4a::/64",
"2001:4860:4801:4b::/64",
"2001:4860:4801:4c::/64",
"2001:4860:4801:50::/64",
"2001:4860:4801:51::/64",
"2001:4860:4801:52::/64",
"2001:4860:4801:53::/64",
"2001:4860:4801:54::/64",
"2001:4860:4801:55::/64",
"2001:4860:4801:56::/64",
"2001:4860:4801:60::/64",
"2001:4860:4801:61::/64",
"2001:4860:4801:62::/64",
"2001:4860:4801:63::/64",
"2001:4860:4801:64::/64",
"2001:4860:4801:65::/64",
"2001:4860:4801:66::/64",
"2001:4860:4801:67::/64",
"2001:4860:4801:68::/64",
"2001:4860:4801:69::/64",
"2001:4860:4801:6a::/64",
"2001:4860:4801:6b::/64",
"2001:4860:4801:6c::/64",
"2001:4860:4801:6d::/64",
"2001:4860:4801:6e::/64",
"2001:4860:4801:6f::/64",
"2001:4860:4801:70::/64",
"2001:4860:4801:71::/64",
"2001:4860:4801:72::/64",
"2001:4860:4801:73::/64",
"2001:4860:4801:74::/64",
"2001:4860:4801:75::/64",
"2001:4860:4801:76::/64",
"2001:4860:4801:77::/64",
"2001:4860:4801:78::/64",
"2001:4860:4801:79::/64",
"2001:4860:4801:80::/64",
"2001:4860:4801:81::/64",
"2001:4860:4801:82::/64",
"2001:4860:4801:83::/64",
"2001:4860:4801:84::/64",
"2001:4860:4801:85::/64",
"2001:4860:4801:86::/64",
"2001:4860:4801:87::/64",
"2001:4860:4801:88::/64",
"2001:4860:4801:90::/64",
"2001:4860:4801:91::/64",
"2001:4860:4801:92::/64",
"2001:4860:4801:93::/64",
"2001:4860:4801:94::/64",
"2001:4860:4801:95::/64",
"2001:4860:4801:96::/64",
"2001:4860:4801:a0::/64",
"2001:4860:4801:a1::/64",
"2001:4860:4801:a2::/64",
"2001:4860:4801:a3::/64",
"2001:4860:4801:a4::/64",
"2001:4860:4801:a5::/64",
"2001:4860:4801:c::/64",
"2001:4860:4801:f::/64",
"192.178.5.0/27",
"192.178.6.0/27",
"192.178.6.128/27",
"192.178.6.160/27",
"192.178.6.192/27",
"192.178.6.32/27",
"192.178.6.64/27",
"192.178.6.96/27",
"34.100.182.96/28",
"34.101.50.144/28",
"34.118.254.0/28",
"34.118.66.0/28",
"34.126.178.96/28",
"34.146.150.144/28",
"34.147.110.144/28",
"34.151.74.144/28",
"34.152.50.64/28",
"34.154.114.144/28",
"34.155.98.32/28",
"34.165.18.176/28",
"34.175.160.64/28",
"34.176.130.16/28",
"34.22.85.0/27",
"34.64.82.64/28",
"34.65.242.112/28",
"34.80.50.80/28",
"34.88.194.0/28",
"34.89.10.80/28",
"34.89.198.80/28",
"34.96.162.48/28",
"35.247.243.240/28",
"66.249.64.0/27",
"66.249.64.128/27",
"66.249.64.160/27",
"66.249.64.224/27",
"66.249.64.32/27",
"66.249.64.64/27",
"66.249.64.96/27",
"66.249.65.0/27",
"66.249.65.128/27",
"66.249.65.160/27",
"66.249.65.192/27",
"66.249.65.224/27",
"66.249.65.32/27",
"66.249.65.64/27",
"66.249.65.96/27",
"66.249.66.0/27",
"66.249.66.128/27",
"66.249.66.160/27",
"66.249.66.192/27",
"66.249.66.224/27",
"66.249.66.32/27",
"66.249.66.64/27",
"66.249.66.96/27",
"66.249.68.0/27",
"66.249.68.128/27",
"66.249.68.32/27",
"66.249.68.64/27",
"66.249.68.96/27",
"66.249.69.0/27",
"66.249.69.128/27",
"66.249.69.160/27",
"66.249.69.192/27",
"66.249.69.224/27",
"66.249.69.32/27",
"66.249.69.64/27",
"66.249.69.96/27",
"66.249.70.0/27",
"66.249.70.128/27",
"66.249.70.160/27",
"66.249.70.192/27",
"66.249.70.224/27",
"66.249.70.32/27",
"66.249.70.64/27",
"66.249.70.96/27",
"66.249.71.0/27",
"66.249.71.128/27",
"66.249.71.160/27",
"66.249.71.192/27",
"66.249.71.224/27",
"66.249.71.32/27",
"66.249.71.64/27",
"66.249.71.96/27",
"66.249.72.0/27",
"66.249.72.128/27",
"66.249.72.160/27",
"66.249.72.192/27",
"66.249.72.224/27",
"66.249.72.32/27",
"66.249.72.64/27",
"66.249.72.96/27",
"66.249.73.0/27",
"66.249.73.128/27",
"66.249.73.160/27",
"66.249.73.192/27",
"66.249.73.224/27",
"66.249.73.32/27",
"66.249.73.64/27",
"66.249.73.96/27",
"66.249.74.0/27",
"66.249.74.128/27",
"66.249.74.160/27",
"66.249.74.192/27",
"66.249.74.32/27",
"66.249.74.64/27",
"66.249.74.96/27",
"66.249.75.0/27",
"66.249.75.128/27",
"66.249.75.160/27",
"66.249.75.192/27",
"66.249.75.224/27",
"66.249.75.32/27",
"66.249.75.64/27",
"66.249.75.96/27",
"66.249.76.0/27",
"66.249.76.128/27",
"66.249.76.160/27",
"66.249.76.192/27",
"66.249.76.224/27",
"66.249.76.32/27",
"66.249.76.64/27",
"66.249.76.96/27",
"66.249.77.0/27",
"66.249.77.128/27",
"66.249.77.160/27",
"66.249.77.192/27",
"66.249.77.224/27",
"66.249.77.32/27",
"66.249.77.64/27",
"66.249.77.96/27",
"66.249.78.0/27",
"66.249.78.32/27",
"66.249.79.0/27",
"66.249.79.128/27",
"66.249.79.160/27",
"66.249.79.192/27",
"66.249.79.224/27",
"66.249.79.32/27",
"66.249.79.64/27",
"66.249.79.96/27"
]
- name: not-googlebot
expression:
all:
- userAgent.matches("\\+http\\://www\\.google\\.com/bot\\.html")
- fcrdns.check("\\.googlebot\\.com$")
action: WEIGH
weight:
adjust: 5
Loading
Loading