Added nix for developer setup (#1379) - Asking for feedback

.envrc

@@ -0,0 +1,8 @@
+# we have checks to account for first time setup
+if [ -f ".env" ]; then
+  dotenv
+fi
+if [ -f "$HOME/.config/nix/nix.conf" ]; then
+  echo "Loading nix shell environment."
+  use flake
+fi

.github/workflows/actions-gcp.yaml

@@ -25,7 +25,7 @@ jobs:
 
     services:
       postgres:
-        image: postgres:12
+        image: postgres:16
         env:
           POSTGRES_USER: postgres
           POSTGRES_PASSWORD: postgres
@@ -44,10 +44,10 @@ jobs:
         with:
           fetch-depth: 0
 
-      - name: Set up Python 3.11
+      - name: Set up Python 3.13
         uses: actions/setup-python@v5
         with:
-          python-version: 3.11
+          python-version: 3.13
 
       - uses: actions/cache@v4
         with:
@@ -133,10 +133,10 @@ jobs:
         run: |
           git fetch --depth=1 origin +refs/tags/*:refs/tags/* || true
 
-      - name: Set up Python 3.11
+      - name: Set up Python 3.13
         uses: actions/setup-python@v5
         with:
-          python-version: 3.11
+          python-version: 3.13
 
       - name: Install Python dependencies
         run: |

.github/workflows/actions.yml

@@ -15,7 +15,7 @@ jobs:
 
     services:
       postgres:
-        image: postgres:12
+        image: postgres:16
         env:
           POSTGRES_USER: postgres
           POSTGRES_PASSWORD: postgres
@@ -34,10 +34,10 @@ jobs:
         with:
           fetch-depth: 0
 
-      - name: Set up Python 3.11
+      - name: Set up Python 3.13
         uses: actions/setup-python@v5
         with:
-          python-version: 3.11
+          python-version: 3.13
 
       - uses: actions/cache@v4
         with:
@@ -85,10 +85,10 @@ jobs:
         run: |
           git fetch --depth=1 origin +refs/tags/*:refs/tags/* || true
 
-      - name: Set up Python 3.11
+      - name: Set up Python 3.13
         uses: actions/setup-python@v5
         with:
-          python-version: 3.11
+          python-version: 3.13
 
       - name: Install Python dependencies
         run: |

.gitignore

@@ -147,7 +147,7 @@ media
 python.log
 
 # direnv
-.envrc
+.direnv
 .local_env
 
 .DS_Store

.pre-commit-config.yaml

@@ -1,9 +1,14 @@
 default_language_version:
-  python: python3.11
+  python: python3.13
 
 exclude: .*migrations\/.*|static\/img\/.*|static\/animations\/.*|static\/js\/boost-gecko\/.*|kube\/boost\/templates\/.*\.yaml
 
 repos:
+  - repo: https://github.com/adamchainz/django-upgrade
+    rev: "1.27.0"
+    hooks:
+      - id: django-upgrade
+        args: [--target-version, "5.2"]   # Replace with Django version
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v5.0.0
     hooks:

README.md

@@ -6,63 +6,75 @@ A Django based website that will power a new Boost website. See the [documentati
 
 Links:
 
-- https://www.stage.boost.cppalliance.org/ - staging
+- https://stage.boost.org/ - staging
 - https://www.boost.org/ - production
 
 ---
 
 ## Local Development Setup
 
-This project will use Python 3.11, Docker, and Docker Compose.
-
-Instructions to install those packages are included in [development_setup_notes.md](docs/development_setup_notes.md).
-
-**NOTE**: All of these various `docker compose` commands, along with other helpful
-developer utility commands, are codified in our `justfile` and can be ran with
-less typing.
-
-You will need to install `just`, by [following the documentation](https://just.systems/man/en/)
-
-**Environment Variables**: Copy file `env.template` to `.env` and adjust values to match your local environment. See [Environment Variables](docs/env_vars.md) for more information.
-
-```shell
-$ cp env.template .env
-```
-
-**NOTE**: Double check that the exposed port assigned to the PostgreSQL
-container does not clash with a database or other server you have running
-locally.
-
-Then run:
+This project will use Python 3.13, Docker, and Docker Compose.
+
+There are two options for development setups, nix and native. The native setup doesn't require nix to be installed. The nix-based setup works in a similar way to a python venv, with everything encapsulated for the project. Note: it is not a vm. It handles installation of all of the application and development environment's dependencies automatically:
+
+- The python version relevant to the project
+- awscli
+- gdk
+- just
+- opentofu
+- nodejs
+- yarn
+- asciidoctor
+- asciidoctor-boost gem
+- pre-commit
+- black
+- isort
+- pip-tools
+
+
+1. Development System setup
+   1. Give your ssh key to Sam so he can add it to the boost.cpp.al server.
+   1. Basic Setup Options:
+      1. [Nix-based](docs/development_setup_notes_nix.md)
+      1. [Native](docs/development_setup_notes_native.md)
+   1. [Initial Data Setup](docs/first_time_data_import.md)
+   1. [Allauth Social Login Setup](docs/allauth_setup.md)
+1. System Concepts
+   1. [GitHub Data Syncing](docs/syncing_data_with_github.md)
+   1. [Rendered Content Caching](docs/caching_rendered_content.md)
+   1. [Static Content from S3](docs/static_content.md)
+   1. ['manage' Commands](docs/commands.md)
+   1. [API](docs/api.md)
+   1. [RSS Feeds](docs/rss_feeds.md)
+   1. [Env Vars](docs/env_vars.md)
+   1. [Calendar](docs/calendar.md)
+   1. [News](docs/news.md)
+1. Instructional
+   1. [Dependency Updates](docs/dependencies.md)
+   1. [Release Reports](docs/release_reports.md)
+   1. [User Management](docs/user_management.md)
+   1. [Non-Dev Server Allauth Setup](docs/non-dev-server-allauth-setup.md)
+   1. [Admin Features](docs/admin.md)
+   1. [Mailing List Setup](docs/mailing_list.md)
+
+After going through the "Development System setup" steps above to create the Docker image, install dependencies, and start the services in `docker-compose.yml`, run:
 
 ```shell
 # start our services (and build them if necessary)
 $ docker compose up
 
 # to create database migrations
-$ docker compose run --rm web python manage.py makemigrations
+$ just makemigrations
 
 # to run database migrations
-$ docker compose run --rm web python manage.py migrate
+$ just migrate
 
 # to create a superuser
-$ docker compose run --rm web python manage.py createsuperuser
+$ just manage createsuperuser
 ```
 
-This will create the Docker image, install dependencies, start the services
-defined in `docker-compose.yml`, and start the webserver.
-
 styles.css is still missing in a local docker-compose environment. Steps to add it:
 
-```
-# One-time setup
-curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.0/install.sh | bash
-. ~/.bashrc
-nvm install 20
-nvm use 20
-npm install -g yarn
-```
-
 ```
 # Each time - rebuild styles.css
 yarn
@@ -90,13 +102,13 @@ $ docker compose down
 To run the tests, execute:
 
 ```shell
-$ docker compose run --rm web pytest
+$ just test
 ```
 
 or run:
 
 ```shell
-$ just test
+$ docker compose run --rm web pytest
 ```
 
 ## Yarn and Tailwind
@@ -125,23 +137,10 @@ currently generated by `docker compose build` and is included in the Docker imag
 
 ---
 
-## Generating Local Data
-
-To **add real Boost libraries and sync all the data from GitHub and S3**, set appropriate values in your new .env file according to [Environment Variables](docs/env_vars.md) for `GITHUB_TOKEN`, `STATIC_CONTENT_AWS_ACCESS_KEY_ID`, `STATIC_CONTENT_AWS_SECRET_ACCESS_KEY`, `STATIC_CONTENT_BUCKET_NAME`, `STATIC_CONTENT_REGION`, `STATIC_CONTENT_AWS_S3_ENDPOINT_URL` then run:
-
-```bash
-docker compose run --rm web python manage.py update_libraries --local
-```
-
-Those values can be gotten from another developer in the `#boost-website` Slack channel.
-
-The `--local` flag speeds up the command a lot by excluding the retrieval of data you generally don't need. For more information, see `update_libraries` in [Management Commands](docs/commands.md).
+## Setting up Mailman/Hyperkitty locally
 
-Then as a superuser log into the admin interface, go to "Versions" and click on the "import new releases" button in the top right.
+Note: This is optional - not required for the website to function, here for reference in case a need arises.
 
----
-
-## Setting up Mailman/Hyperkitty locally
 ``shell
 sudo apt-get install sassc
 git clone [email protected]:mailman/hyperkitty.git
@@ -152,23 +151,15 @@ pip install psycopg2-binary
 ``
 change settings.py to use postgres database:
         'ENGINE': 'django.db.backends.postgresql_psycopg2',
-Update database values in settings to use the same host, user, password, and the database name as in the .env file value for `HYPERKITTY_DATABASE_NAME`.
+Update database values in settings to use the same host, user, password, and the database name as in the .env file value for `HYPERKITTY_DATABASE_NAME` (`lists_production_web` by default).
 
 run `django-admin migrate --pythonpath example_project --settings settings`
 
-Give your ssh key to Sam so he can add it to the boost.cpp.al server, and then download the mailman db archive and cp the sql to the docker container
-```shell
-scp {user}@staging-db1.boost.cpp.al:/tmp/lists_stage_web.staging-db1-2.2025-02-06-08-00-01.sql.gz .
-docker cp lists_stage_web.staging-db1-2.2025-02-06-08-00-01.sql website-v2-web-1:/lists_stage_web.staging-db1-2.2025-02-06-08-00-01.sql
-docker exec -it website-v2-web-1 /bin/bash
-psql -U postgres -W hyperkitty_db < /lists_stage_web.staging-db1-2.2025-02-06-08-00-01.sql
-```
-
 ## Syncing EmailData Locally
 
 To work with mailinglist data locally, the django application expects to be
 able to query a copy of the hyperkitty database from HYPERKITTY_DATABASE_NAME.
-Then, the `sync_mailinglist_stats` management command can be run.
+Then, `just manage sync_mailinglist_stats` management command can be run.
 
 ## Deploying
 
@@ -190,17 +181,6 @@ We use [pre-commit hooks](https://pre-commit.com/) to check code for style, synt
 | [Ruff](https://github.com/charliermarsh/ruff) | Wrapper around `flake8` and `isort`, among other linters |
 | [Djhtml](https://github.com/rtts/djhtml) | Auto-formats Django templates |
 
-### Setup and Usage
-
-| Description | Command |
-| ---- | ------- |
-| 1. Install the `pre-commit` package using `pip` | `pip install pre-commit` |
-| 2. Install our list of pre-commit hooks locally | `pre-commit install` |
-| 3. Run all hooks for changed files before commit | `pre-commit run` |
-| 4. Run specific hook before commit | `pre-commit run {hook}` |
-| 5. Run hooks for all files, even unchanged ones | `pre-commit run --all-files` |
-| 6. Commit without running pre-commit hooks | `git commit -m "Your commit message" --no-verify` |
-
 Example commands for running specific hooks:
 
 | Hook | Example |

core/calendar.py

@@ -1,4 +1,4 @@
-import datetime
+from datetime import datetime, timezone
 from django.conf import settings
 import requests
 from collections import defaultdict
@@ -20,10 +20,10 @@ def get_calendar(min_time=None, single_events=True, order_by="startTime"):
     https://developers.google.com/calendar/api/v3/reference/events/list
     """
     if not min_time:
-        min_time = (
-            datetime.datetime.utcnow().isoformat() + "Z"
-        )  # 'Z' indicates UTC time
-
+        # 'Z' indicates UTC time
+        # we replace +00:00 with Z because google no longer seems to be fully rfc3339
+        # compliant for this parameter, even with a valid format ending with +00:00
+        min_time = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
     url = f"https://www.googleapis.com/calendar/v3/calendars/{settings.BOOST_CALENDAR}/events?key={settings.CALENDAR_API_KEY}&timeMin={min_time}&singleEvents={single_events}&orderBy={order_by}"
 
     headers = {"Accept": "application/json"}

core/migrations/0003_sitesettings_and_more.py

@@ -28,7 +28,7 @@ class Migration(migrations.Migration):
         migrations.AddConstraint(
             model_name="sitesettings",
             constraint=models.CheckConstraint(
-                check=models.Q(("id", 1)), name="core_sitesettings_single_instance"
+                condition=models.Q(("id", 1)), name="core_sitesettings_single_instance"
             ),
         ),
         migrations.AlterModelOptions(

core/models.py

@@ -72,7 +72,7 @@ class Meta:
             # check constraint to only allow id=1 to exist
             models.CheckConstraint(
                 name="%(app_label)s_%(class)s_single_instance",
-                check=models.Q(id=1),
+                condition=models.Q(id=1),
             ),
         ]
         verbose_name_plural = "Site Settings"