Skip to content

Commit 80672e6

Browse files
daveoconnordaveoconnor
committed
Added nix for developer setup (#1379)
1 parent b74c1c9 commit 80672e6

File tree

7 files changed

+351
-11
lines changed

7 files changed

+351
-11
lines changed

‎.envrc

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# we have checks to account for first time setup
2+
if [ -f ".env" ]; then
3+
dotenv
4+
fi
5+
if [ -f "$HOME/.config/nix/nix.conf" ]; then
6+
echo "Entering nix shell environment."
7+
use flake
8+
fi

‎.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,7 @@ media
147147
python.log
148148

149149
# direnv
150-
.envrc
150+
.direnv
151151
.local_env
152152

153153
.DS_Store

‎docs/nix_based_setup.md

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Boost.org Website
2+
3+
## Overview
4+
5+
A Django based website that will power a new Boost website. See the [documentation](./docs/README.md) for more information about maintaining this project.
6+
7+
Links:
8+
9+
- https://www.stage.boost.cppalliance.org/ - staging
10+
- https://www.boost.org/ - production
11+
12+
---
13+
14+
## Local Development Setup
15+
16+
This project uses Python 3.11, Docker, and Docker Compose.
17+
18+
This document describes how to set up a development environment using Nix, which is a package manager that allows for reproducible builds and development environments, like a better encapsulated declarative cross-platform Homebrew.
19+
20+
For a basic rundown on Nix, this video could be useful https://www.youtube.com/watch?v=yQwW8dkuHqw
21+
22+
1. Install the following according to the instructions for your platform if not already installed:
23+
1. Direnv - https://direnv.net/docs/installation.html (don't install OS packaged version, must be >= 2.35.0) and then configure your shell to add the hook as per the direnv docs.
24+
2. Docker Engine
25+
* Linux - https://docs.docker.com/engine/install/
26+
* MacOS - https://orbstack.dev/ or https://github.com/abiosoft/colima ?
27+
* Windows - ?
28+
3. Just - https://just.systems/man/en/packages.html
29+
4. Nix - https://nixos.org/download/ (multi-user, y to all options)
30+
2. Clone this https://github.com/boostorg/website-v2.git repository to your machine.
31+
3. cd into the repository directory.
32+
4. In a terminal run `just bootstrap` in the root of the checked out repository to install the necessary development dependencies and generate the .env file.
33+
5. Update the generated .env file with the necessary environment variables. Where you can't retrieve these yourself, you can ask someone for some in #boost-website on the slack server at https://ccplang.slack.com. The minimum that must be set is:
34+
* GITHUB_TOKEN - a personal access token for the GitHub API, from your profile
35+
* STATIC_CONTENT_AWS_ACCESS_KEY_ID - ask for this
36+
* STATIC_CONTENT_AWS_SECRET_ACCESS_KEY - ask for this
37+
* STATIC_CONTENT_BUCKET_NAME - ask for this
38+
* STATIC_CONTENT_REGION - ask for this
39+
* STATIC_CONTENT_AWS_S3_ENDPOINT_URL - ask for this
40+
5. Run `just setup` to build services, and build the JS and CSS assets. If docker fails with permissions errors, reboot your machine.
41+
6. Run `just load_production_data` to download live data from the backup server.
42+
7. Run `docker compose up` to start the server.
43+
44+
### Social Login with django-allauth
45+
46+
Follow these instructions to use the social logins through django-allauth on your local machine.
47+
48+
See https://testdriven.io/blog/django-social-auth/ for more information.
49+
50+
#### Github
51+
- Go to https://github.com/settings/applications/new and add a new OAuth application
52+
- Set `http://localhost:8000` as the Homepage URL
53+
- Set `http://localhost:8000/accounts/github/login/callback/` as the Callback URL
54+
- Click whether you want to enable the device flow
55+
- <img src="https://user-images.githubusercontent.com/2286304/252841283-9a846c68-46bb-4dac-8d1e-d35270c09f1b.png" alt="The GitHub screen that registers a new OAuth app" width="400">
56+
- On completion copy the Client ID and Client Secret to the `.env` file as values of `GITHUB_OAUTH_CLIENT_ID` and `GITHUB_OAUTH_CLIENT_SECRET`.
57+
- Run `direnv allow` and restart your docker containers.
58+
59+
Setup should be complete and you should be able to see an option to "Use Github" on the sign up page.
60+
61+
To test the flow including authorizing Github for the Boost account, log into your GitHub account settings and click **Applications** in the left menu. Find the "Boost" authorization and delete it. The next time you log into Boost with this GitHub account, you will have to re-authorize it.
62+
63+
<img src="https://user-images.githubusercontent.com/2286304/204642346-8b269aaf-4693-4351-9474-0a998b97689c.png" alt="The 'Authorized OAuth Apps' tab in your GitHub Applications" width="400">
64+
65+
This setup process is not something that can currently be automated through terraform because of a lack of relevant Github API endpoints to create Oauth credentials.
66+
67+
#### Google
68+
69+
More detailed instructions at:
70+
71+
https://docs.allauth.org/en/latest/socialaccount/providers/google.html
72+
73+
1. Update the `.env` file with values for:
74+
1. `TF_VAR_google_cloud_email` (the email address of your Google Cloud account)
75+
2. `TF_VAR_google_organization_domain` (usually the domain of your Google Cloud account, e.g. "boost.org" if you will be using a @boost.org email address)
76+
3. `TF_VAR_google_cloud_project_name` (optional, default: localboostdev) - needs to change if destroyed and a setup is needed within 30 days
77+
2. Run `just development-tofu-init` to initialize tofu.
78+
3. Run `just development-tofu-plan` to confirm the planned changes.
79+
4. Run `just development-tofu-apply` to apply the changes.
80+
5. Go to https://console.developers.google.com/
81+
1. Search for the newly created project, named "Boost Development" (ID: localboostdev by default).
82+
2. Type "credentials" in the search input at the top of the page.
83+
3. Select "Credentials" under "APIs & Services".
84+
1. Click "+ CREATE CREDENTIALS"
85+
2. Select "OAuth Client ID"
86+
3. Select Application Type: "Web application"
87+
4. Name: "Boost Development" (arbitrary)
88+
5. For "Authorized Javascript Origins" use:`http://localhost:8000`
89+
6. For "Authorized Redirect URIs" use:
90+
* `http://localhost:8000/accounts/google/login/callback/`
91+
* `http://localhost:8000/accounts/google/login/callback/?flowName=GeneralOAuthFlow`
92+
7. Save
93+
6. From the page that's displayed, update the `.env` file with values for the following:
94+
- `GOOGLE_OAUTH_CLIENT_ID` should be similar to "k235bn2b1l1(...)asdsk.apps.googleusercontent.com"
95+
- `GOOGLE_OAUTH_CLIENT_SECRET` should be similar to "LAJACO(...)KLAI612ANAD"
96+
7. Run `docker compose down && docker compose up` and restart your docker containers.
97+
98+
Point 5 above can not be automated through terraform because of a lack of relevant Google Cloud API endpoints to create Oauth credentials.
99+
100+
Setup should be complete and you should be able to see an option to "Use Google" on the sign up page.
101+
102+
#### Additional Notes on allauth login flows:
103+
**Working locally**: If you need to run through the login flows multiple times, create a superuser so you can log into the admin. Then, log into the admin and delete your "Social Account" from the admin. This will test a fresh connection to GitHub for your logged-in GitHub user.
104+
105+
## Syncing EmailData Locally (optional)
106+
107+
To work with mailinglist data locally, the django application expects to be
108+
able to query a copy of the hyperkitty database from HYPERKITTY_DATABASE_NAME.
109+
Then, `just manage sync_mailinglist_stats` management command can be run.
110+
111+
## Debugging
112+
For local development there is Django Debug Toolbar, and the option to set a debugger.
113+
114+
In your env:
115+
- Django Debug Toolbar, enabled by default, can be disabled by setting DEBUG_TOOLBAR=False
116+
- IDE Debugging, disabled by default, can be enabled by uncommenting `PYTHONBREAKPOINT` in your .env file.
117+
118+
### Set Up Pycharm
119+
You can set up your IDE with a new "Python Debug Server" configuration as:
120+
121+
<img src="images/pycharm_debugger_settings.png" alt="PyCharm Debugger Settings" width="400">
122+
123+
### Debugger Usage
124+
To use the debugger add `breakpoint()` on a line in the code before you want to start debugging and then add breakpoints by clicking on the gutter. The debugger will stop at these point, you can then step/inspect the variables.
125+
126+
127+
## Troubleshooting
128+
129+
### Docker
130+
Keep in mind if there are issues with docker that the host docker daemon on your machine and the docker daemon in the nix setup may not match. It's a good idea to keep both up to date.
131+
132+
### Direnv
133+
when you switch to the directory you may see direnv exporting a bunch of environment variables as below.
134+
135+
The installer configures direnv to suppress those but it's a recent configuration option, so may be worth checking for an update if you see them.
136+
137+
## Disk space
138+
Should you find you're running short on disk space, to delete previous versioned store data you can run `nix-collect-garbage -d`. Reentering the directory will then reinstall all the current dependencies again. It's probably a good idea to run that periodically.
139+
140+
```shell
141+
direnv: export +ALLOWED_HOSTS +AR +AS...
142+
```

‎flake.lock

Lines changed: 61 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎flake.nix

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
{
2+
description = "Boost.org development environment.";
3+
4+
inputs = {
5+
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
6+
flake-utils.url = "github:numtide/flake-utils";
7+
};
8+
9+
outputs = { self, nixpkgs, flake-utils, ... }@inputs:
10+
flake-utils.lib.eachDefaultSystem (system:
11+
let
12+
pkgs = import nixpkgs {
13+
inherit system;
14+
};
15+
# https://nixos.wiki/wiki/Google_Cloud_SDK
16+
gdk = pkgs.google-cloud-sdk.withExtraComponents( with pkgs.google-cloud-sdk.components; [
17+
gke-gcloud-auth-plugin
18+
]);
19+
# Install a Ruby gem from rubygems.org
20+
asciidoctorBoostGem = pkgs.stdenv.mkDerivation rec {
21+
pname = "asciidoctor-boost";
22+
version = "0.1.7";
23+
sha = "ce139448812a9848219ce4cdb521c83c16009406a9d16efbc90bb24e94a46c24";
24+
25+
src = pkgs.fetchurl {
26+
url = "https://rubygems.org/downloads/${pname}-${version}.gem";
27+
sha256 = "${sha}";
28+
};
29+
dontUnpack = true;
30+
nativeBuildInputs = [ pkgs.ruby ];
31+
buildPhase = "true"; # Nothing to compile.
32+
installPhase = ''
33+
# Create a temporary gem directory
34+
mkdir -p $out
35+
# Set GEM_HOME to install gems locally under $out.
36+
export GEM_HOME=$out
37+
# Install the gem into GEM_HOME.
38+
${pkgs.ruby}/bin/gem install ${src} --no-document --ignore-dependencies
39+
'';
40+
meta = {
41+
description = "Asciidoctor Boost Ruby Gem installed from rubygems.org";
42+
homepage = "https://rubygems.org/gems/asciidoctor-boost";
43+
license = "BSL-1.0";
44+
};
45+
};
46+
47+
in {
48+
devShells.default = pkgs.mkShell {
49+
buildInputs = with pkgs; [
50+
# general system
51+
# e.g. this could contain docker client if we wanted that to be consistent,
52+
# though we need the daemon on the host anyway so it's redundant
53+
# general project
54+
awscli
55+
gdk
56+
just
57+
opentofu
58+
# frontend
59+
nodejs_22 # matches Dockerfile, due for upgrade?
60+
yarn
61+
# backend
62+
asciidoctor
63+
asciidoctorBoostGem
64+
pre-commit
65+
python311 # matches Dockerfile, due for upgrade?
66+
python311.pkgs.black
67+
python311.pkgs.isort
68+
python311.pkgs.pip-tools
69+
];
70+
# Host system installation workflow goes into the bootstrap justfile target.
71+
# Project specific installation and execution workflow should go here.
72+
shellHook = ''
73+
if [ ! -f .git/hooks/pre-commit ]; then
74+
pre-commit install
75+
fi
76+
if [ ! -d .venv ]; then
77+
python3.11 -m venv .venv
78+
. .venv/bin/activate
79+
pip install -r requirements.txt -r requirements-dev.txt
80+
else
81+
. .venv/bin/activate
82+
fi
83+
if [ ! -f .env ]; then
84+
cp env.template .env
85+
echo ".env created, you should update its contents"
86+
fi
87+
# google cloud login
88+
gcloud auth list --format="value(account)" | grep -q . || {
89+
echo "Not logged in. Running gcloud auth login..."
90+
gcloud auth login
91+
}
92+
'';
93+
};
94+
}
95+
);
96+
}

‎justfile

Lines changed: 39 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,43 @@ ENV_FILE := ".env"
1212
# ----
1313

1414
@bootstrap: ## installs/updates all dependencies
15-
#!/usr/bin/env bash
16-
set -euo pipefail
17-
if [ ! -f "{{ENV_FILE}}" ]; then
18-
echo "{{ENV_FILE}} created"
19-
cp env.template {{ENV_FILE}}
15+
command -v direnv >/dev/null 2>&1 || { echo >&2 "Direnv is required but not installed. see: https://direnv.net/docs/installation.html - Aborting."; exit 1; }
16+
command -v nix >/dev/null 2>&1 || { echo >&2 "Nix is required but not installed. see: https://nixos.org/download.html - Aborting."; exit 1; }
17+
command -v just >/dev/null 2>&1 || { echo >&2 "Just is required but not installed. see: https://just.systems/man/en/packages.html - Aborting."; exit 1; }
18+
command -v docker >/dev/null 2>&1 || { echo >&2 "Docker is required but not installed. see: docs for links - Aborting."; exit 1; }
19+
20+
shell_name=$(basename "$SHELL") && \
21+
echo $shell_name && \
22+
if [ "$shell_name" = "zsh" ] && command -v zsh >/dev/null; then \
23+
zsh -i -c 'echo ${precmd_functions} | grep -q _direnv_hook' || { echo "❌ direnv hook is NOT installed in Zsh"; exit 1; }; \
24+
elif ([ "$shell_name" = "pwsh" ] || [ "$shell_name" = "powershell" ]) && command -v "$shell_name" >/dev/null; then \
25+
"$shell_name" -NoProfile -Command '$function:prompt.ToString() | grep -q direnv' || { echo "❌ direnv hook is NOT installed in PowerShell"; exit 1; }; \
26+
else \
27+
echo "ℹ️ Unsupported shell for checking direnv hook: $shell_name. Ensure you have the direnv shell hook eval set up correctly if there are problems."; \
2028
fi
21-
docker compose --file {{COMPOSE_FILE}} build --force-rm
29+
30+
if [ ! -d $HOME/.config/direnv/direnv.toml ]; then \
31+
mkdir -p $HOME/.config/direnv; \
32+
printf "[global]\nhide_env_diff = true\nload_dotenv = true\n" > $HOME/.config/direnv/direnv.toml; \
33+
fi
34+
if [ ! -d $HOME/.config/nix ]; then \
35+
mkdir -p $HOME/.config/nix; \
36+
printf "experimental-features = nix-command flakes\n" > $HOME/.config/nix/nix.conf; \
37+
fi
38+
# check if the docker group exists, create if not
39+
if [ ! $(getent group docker) ]; then \
40+
echo "ℹ️ Adding docker group..."; \
41+
sudo groupadd docker; \
42+
fi
43+
44+
# check if user is in docker group, add if not
45+
if [ $(id -Gn | grep -c docker) -eq 0 ]; then \
46+
echo "ℹ️ Adding docker group"; \
47+
sudo usermod -aG docker $USER; \
48+
echo "ℹ️ Added docker user. Please close the shell and open a new one."; \
49+
fi
50+
echo "Bootstrapping complete, update your .env and run 'just setup'"
51+
echo "If you have issues with docker permissions running just setup try restarting your machine."
2252

2353
@rebuild: ## rebuilds containers
2454
docker compose kill
@@ -31,7 +61,7 @@ ENV_FILE := ".env"
3161

3262
@build: ## builds containers
3363
docker compose pull
34-
DOCKER_BUILDKIT=1 docker compose build
64+
docker compose build
3565

3666
@cibuild: ## invoked by continuous integration servers to run tests
3767
python -m pytest
@@ -49,6 +79,8 @@ alias shell := console
4979
@setup: ## sets up a project to be used for the first time
5080
docker compose --file {{COMPOSE_FILE}} build --force-rm
5181
docker compose --file docker-compose.yml run --rm web python manage.py migrate --noinput
82+
npm install
83+
npm run build
5284

5385
@test_pytest: ## runs pytest
5486
-docker compose run --rm -e DEBUG_TOOLBAR="False" web pytest -s --create-db

0 commit comments

Comments
 (0)