v1.21.0: Minfilia Warde

[Xe]

Please, be at ease. You are among friends here.

In this release, Anubis becomes internationalized, gains the ability to use system load as input to issuing challenges, finally fixes the "invalid response" after "success" bug, and more! Please read these notes before upgrading as the changes are big enough that administrators should take action to ensure that the upgrade goes smoothly.

Big ticket changes

The biggest change is that the "invalid response" after "success" bug is now finally fixed for good by totally rewriting how Anubis' challenge issuance flow works. Instead of generating challenge strings from request metadata (under the assumption that the values being compared against are stable), Anubis now generates random data for each challenge. This data is stored in the active storage backend for up to 30 minutes. This also fixes #746 and other similar instances of this issue.

In order to reduce confusion, the "Success" interstitial that shows up when you pass a proof of work challenge has been removed.

Storage

Anubis now is able to store things persistently in memory, on the disk, or in Valkey (this includes other compatible software). By default Anubis uses the in-memory backend. If you have an environment with mutable storage (even if it is temporary), be sure to configure the bbolt storage backend.

Localization

Anubis now supports localized responses. Locales can be added in lib/localization/locales/. This release includes support for the following languages:

• Brazilian Portugese
• Chinese (Simplified)
• Chinese (Traditional)
• English
• Estonian
• Filipino
• French
• German
• Icelandic
• Italian
• Japanese
• Spanish
• Turkish

If facts or local regulations demand, you can set Anubis default language with the FORCED_LANGUAGE environment variable or the --forced-language command line argument:

FORCED_LANGUAGE=de

Load average

Anubis can dynamically take action based on the system load average, allowing you to write rules like this:

## System load based checks.
# If the system is under high load for the last minute, add weight.
- name: high-load-average
  action: WEIGH
  expression: load_1m >= 10.0 # make sure to end the load comparison in a .0
  weight:
    adjust: 20

# If it is not for the last 15 minutes, remove weight.
- name: low-load-average
  action: WEIGH
  expression: load_15m <= 4.0 # make sure to end the load comparison in a .0
  weight:
    adjust: -10

Something to keep in mind about system load average is that it is not aware of the number of cores the system has. If you have a 16 core system that has 16 processes running but none of them is hogging the CPU, then you will get a load average below 16. If you are in doubt, make your "high load" metric at least two times the number of CPU cores and your "low load" metric at least half of the number of CPU cores. For example:

Kind	Core count	Load threshold
high load	4	8.0
low load	4	2.0
high load	16	32.0
low load	16	8

Also keep in mind that this does not account for other kinds of latency like I/O latency. A system can have its web applications unresponsive due to high latency from a MySQL server but still have that web application server report a load near or at zero.

Other features and fixes

There are a bunch of other assorted features and fixes too:

• Add COOKIE_SECURE option to set the cookie Secure flag
• Sets cookie defaults to use SameSite: None
• Determine the BIND_NETWORK/--bind-network value from the bind address (#677).
• Implement a development container manifest to make contributions easier.
• Fix dynamic cookie domains functionality (#731)
• Add option for custom cookie prefix (#732)
• Make the Open Graph subsystem and DNSBL subsystem use storage backends instead of storing everything in memory by default.
• Allow Common Crawl by default so scrapers have less incentive to scrape
• The bbolt storage backend now runs its cleanup every hour instead of every five minutes.
• Don't block Anubis starting up if Thoth health checks fail.
• A race condition involving opening two challenge pages at once in different tabs causing one of them to fail has been fixed.
• The "Try again" button on the error page has been fixed. Previously it meant "try the solution again" instead of "try the challenge again".
• In certain cases, a user could be stuck with a test cookie that is invalid, locking them out of the service for up to half an hour. This has been fixed with better validation of this case and clearing the cookie.
• Start exposing JA4H fingerprints for later use in CEL expressions.
• Add /healthz route for use in platform-based health checks.

Potentially breaking changes

We try to introduce breaking changes as much as possible, but these are the changes that may be relevant for you as an administrator:

Challenge format change

Previously Anubis did no accounting for challenges that it issued. This means that if Anubis restarted during a client, the client would be able to proceed once Anubis came back online.

During the upgrade to v1.21.0 and when v1.21.0 (or later) restarts with the in-memory storage backend, you may see a higher rate of failed challenges than normal. If this persists beyond a few minutes, open an issue.

If you are using the in-memory storage backend, please consider using a different storage backend.

Systemd service changes

The following potentially breaking change applies to native installs with systemd only:

Each instance of systemd service template now has a unique RuntimeDirectory, as opposed to each instance of the service sharing a RuntimeDirectory. This change was made to avoid the RuntimeDirectory getting nuked any time one of the Anubis instances restarts.

If you configured Anubis' unix sockets to listen on /run/anubis/foo.sock for instance anubis@foo, you will need to configure Anubis to listen on /run/anubis/foo/foo.sock and additionally configure your HTTP load balancer as appropriate.

If you need the legacy behaviour, install this systemd unit dropin:

# /etc/systemd/system/[email protected]/50-runtimedir.conf
[Service]
RuntimeDirectory=anubis

Just keep in mind that this will cause problems when Anubis restarts.

What's Changed

• feat: implement localization system by @lolgzs in feat: implement localization system #716
• fix: determine bind network from bind address by @littlecxm in fix: determine bind network from bind address #714
• Add Brazilian Portuguese translation by @rffontenelle in Add Brazilian Portuguese translation #726
• fix: Dynamic cookie domain not working by @Earl0fPudding in fix: Dynamic cookie domain not working #731
• feat(cmd): Add custom cookie prefix by @Earl0fPudding in feat(cmd): Add custom cookie prefix #732
• build(deps): bump the github-actions group with 2 updates by @dependabot[bot] in build(deps): bump the github-actions group with 2 updates #735
• build(deps): bump the gomod group with 2 updates by @dependabot[bot] in build(deps): bump the gomod group with 2 updates #736
• feat: dev container support by @Xe in feat: dev container support #734
• Fix translations in pt-BR.json by @rffontenelle in Fix translations in pt-BR.json #729
• Set cookies to have the Secure flag default to true by @victorvalenca in Set cookies to have the Secure flag default to true #739
• fix(web/main): remove the success interstitial by @Xe in fix(web/main): remove the success interstitial #745
• feat(localization): Add option for forcing a language by @Earl0fPudding in feat(localization): Add option for forcing a language #742
• fix(run/[email protected]): unique runtimedir per instance by @Xe in fix(run/[email protected]): unique runtimedir per instance #750
• feat(localization): Add German language translation by @Earl0fPudding in feat(localization): Add German language translation #741
• docs: add BotStopper docs from the git repo by @Xe in docs: add BotStopper docs from the git repo #752
• chore(default-config): allowlist common crawl by @Xe in chore(default-config): allowlist common crawl #753
• feat(localization): Add Turkish language translation by @dcelasun in feat(localization): Add Turkish language translation #751
• docs(known-instances): add ebird.org by @SGHFan in docs(known-instances): add ebird.org #755
• feat(lib): use new challenge creation flow by @Xe in feat(lib): use new challenge creation flow #749
• chore(devcontainer): move playwright to its own devcontainer service by @Xe in chore(devcontainer): move playwright to its own devcontainer service #756
• docs(known-instances): Add Duke University, coinhoards.org (and myself) to known instances by @lotharsm in docs(known-instances): Add Duke University, coinhoards.org (and myself) to known instances #757
• fix: make ogtags and dnsbl use the Store instead of memory by @Xe in fix: make ogtags and dnsbl use the Store instead of memory #760
• fix(lib/store/bbolt): use a multi-bucket flow instead of a single bucket flow by @Xe in fix(lib/store/bbolt): use a multi-bucket flow instead of a single bucket flow #761
• fix(lib/store/bbolt): run cleanup every hour instead of every 5 minutes by @Xe in fix(lib/store/bbolt): run cleanup every hour instead of every 5 minutes #762
• docs: remove proof of work branding by @Xe in docs: remove proof of work branding #763
• feat(localization): Update German language translation by @lotharsm in feat(localization): Update German language translation #764
• docs(known-instances): update list of known instances by @lotharsm in docs(known-instances): update list of known instances #767
• feat(localization): Add Traditional Chinese language translation by @xlionjuan in feat(localization): Add Traditional Chinese language translation #759
• feat(lib/policy/expressions): add system load average to bot expression inputs by @Xe in feat(lib/policy/expressions): add system load average to bot expression inputs #766
• build(deps): bump the github-actions group with 2 updates by @dependabot[bot] in build(deps): bump the github-actions group with 2 updates #770
• build(deps): bump github.com/shirou/gopsutil/v4 from 4.25.1 to 4.25.6 in the gomod group by @dependabot[bot] in build(deps): bump github.com/shirou/gopsutil/v4 from 4.25.1 to 4.25.6 in the gomod group #771
• minor typo fix: Update apache.mdx replace nginx with Apache in place by @mihugo in minor typo fix: Update apache.mdx replace nginx with Apache in place #779
• docs(known-instances): update list of known instances by @lotharsm in docs(known-instances): update list of known instances #776
• feat(localization): add Simplified Chinese by @littlecxm in feat(localization): add Simplified Chinese #774
• docs(installation): Clarify information about private keys and multile instances by @StandingPadAnimations in docs(installation): Clarify information about private keys and multile instances #788
• fix(localization): HTML language header and forced-language by @SlyEcho in fix(localization): HTML language header and forced-language #787
• Update apache.mdx by @jzb in Update apache.mdx #784
• feat(localization): add Japanese language translation by @dai in feat(localization): add Japanese language translation #772
• feat(i18n): add Estonian locale by @SlyEcho in feat(i18n): add Estonian locale #783
• Create is.json by @sveinki in Create is.json #780
• feat(localization): Add Italian language translation by @giomba in feat(localization): Add Italian language translation #778
• feat(localization): Add Filipino language by @searinminecraft in feat(localization): Add Filipino language #775
• fix(internal/thoth): don't block Anubis starting if healthcheck fails by @Xe in fix(internal/thoth): don't block Anubis starting if healthcheck fails #794
• feat(blog): incident report for TI-20250709-0001 by @Xe in feat(blog): incident report for TI-20250709-0001 #795
• chore: use nginx-micro to make the docs image 13 MB by @Xe in chore: use nginx-micro to make the docs image 13 MB #796
• docs: update CHANGELOG for language changes by @Xe in docs: update CHANGELOG for language changes #793
• docs(known-instances): update list of known instances by @lotharsm in docs(known-instances): update list of known instances #801
• correct gitea.botPolicies extension to be yaml, not json by @evgeni in correct gitea.botPolicies extension to be yaml, not json #800
• docs(known-instances): add rpmfusion.org and wiki.freepascal.org to known instances by @lotharsm in docs(known-instances): add rpmfusion.org and wiki.freepascal.org to known instances #807
• chore(docs): fix typo in configuration/expressions by @maximelouet in chore(docs): fix typo in configuration/expressions #811
• fix(index.templ) centered-div class usage typo by @ciencia in fix(index.templ) centered-div class usage typo #812
• chore(docs): add link to status page in the footer by @Xe in chore(docs): add link to status page in the footer #814
• chore: release v1.21.0-pre2 by @Xe in chore: release v1.21.0-pre2 #816
• build(deps): bump the gomod group with 3 updates by @dependabot[bot] in build(deps): bump the gomod group with 3 updates #823
• build(deps-dev): bump esbuild from 0.25.5 to 0.25.6 in the npm group by @dependabot[bot] in build(deps-dev): bump esbuild from 0.25.5 to 0.25.6 in the npm group #825
• fix(default-config): disable system load check by default by @Xe in fix(default-config): disable system load check by default #827
• test: add smoke test for git clone by @Xe in test: add smoke test for git clone #828
• test: add git push smoke test by @Xe in test: add git push smoke test #830
• ci(docs): make a new docker image for the docs per commit sha by @Xe in ci(docs): make a new docker image for the docs per commit sha #831
• fix: race conditions, cookie logic, and the try again button by @Xe in fix: race conditions, cookie logic, and the try again button #833
• feat(cmd/anubis): capture ja4h fingerprints by @Xe in feat(cmd/anubis): capture ja4h fingerprints #834
• feat: add Uptime Robot policy definition by @herrbischoff in feat: add Uptime Robot policy definition #838
• fix(localization): fix missing string in template by @littlecxm in fix(localization): fix missing string in template #835
• feat(anubis): add /healthz route to metrics server by @Xe in feat(anubis): add /healthz route to metrics server #843
• chore: release v1.21.0 by @Xe in chore: release v1.21.0 #844

New Contributors

• @lolgzs made their first contribution in feat: implement localization system #716
• @littlecxm made their first contribution in fix: determine bind network from bind address #714
• @rffontenelle made their first contribution in Add Brazilian Portuguese translation #726
• @victorvalenca made their first contribution in Set cookies to have the Secure flag default to true #739
• @dcelasun made their first contribution in feat(localization): Add Turkish language translation #751
• @SGHFan made their first contribution in docs(known-instances): add ebird.org #755
• @xlionjuan made their first contribution in feat(localization): Add Traditional Chinese language translation #759
• @mihugo made their first contribution in minor typo fix: Update apache.mdx replace nginx with Apache in place #779
• @StandingPadAnimations made their first contribution in docs(installation): Clarify information about private keys and multile instances #788
• @jzb made their first contribution in Update apache.mdx #784
• @dai made their first contribution in feat(localization): add Japanese language translation #772
• @sveinki made their first contribution in Create is.json #780
• @giomba made their first contribution in feat(localization): Add Italian language translation #778
• @searinminecraft made their first contribution in feat(localization): Add Filipino language #775
• @evgeni made their first contribution in correct gitea.botPolicies extension to be yaml, not json #800
• @maximelouet made their first contribution in chore(docs): fix typo in configuration/expressions #811
• @ciencia made their first contribution in fix(index.templ) centered-div class usage typo #812
• @herrbischoff made their first contribution in feat: add Uptime Robot policy definition #838

Full Changelog: v1.20.0...v1.21.0

---

This discussion was created from the release v1.21.0: Minfilia Warde.