Merge pull request #300 from edx/dkaplan1/APER-4252_block-robots.txt-on-all-mfes,-idas,-and-edxapp

chore: block robots.txt on all mfes, idas, and edxapp

Author: deborahgu

CHANGELOG.md

@@ -4,6 +4,9 @@
 All notable changes to this project will be documented in this file.
 Add any new changes to the top (right below this line).
 
+- 2025-12-05
+  - Set `NGINX_ROBOT_RULES` to a default of block everything.
+
 - 2025-09-26
   - Moved `EDXAPP_FEATURES_DEFAULT` and `EDXAPP_FEATURES_EXTRA` into top-level settings.
 

playbooks/roles/nginx/defaults/main.yml

@@ -190,7 +190,10 @@ nginx_ecommerce_gunicorn_hosts:
 nginx_credentails_gunicorn_hosts:
   - 127.0.0.1
 
-NGINX_ROBOT_RULES: [ ]
+# Block all crawlers by default
+NGINX_ROBOT_RULES:
+  - agent: "*"
+    disallow: "/"
 NGINX_EDXAPP_EMBARGO_CIDRS: []
 NGINX_P3P_MESSAGE: 'CP="Open edX does not have a P3P policy."'