Skip to content

Commit b692281

Browse files
daveoconnordaveoconnor
committed
Added nix for developer setup (#1379)
1 parent 42b7877 commit b692281

File tree

6 files changed

+316
-8
lines changed

6 files changed

+316
-8
lines changed

‎.envrc

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# we have checks to account for first time setup
2+
if [ -f ".env" ]; then
3+
dotenv
4+
fi
5+
if [ -f "$HOME/.config/nix/nix.conf" ]; then
6+
echo "Entering nix shell environment."
7+
use flake
8+
fi

‎.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,7 @@ media
147147
python.log
148148

149149
# direnv
150-
.envrc
150+
.direnv
151151
.local_env
152152

153153
.DS_Store

‎docs/nix_based_setup.md

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Boost.org Website
2+
3+
## Overview
4+
5+
A Django based website that will power a new Boost website. See the [documentation](./docs/README.md) for more information about maintaining this project.
6+
7+
Links:
8+
9+
- https://www.stage.boost.cppalliance.org/ - staging
10+
- https://www.boost.org/ - production
11+
12+
---
13+
14+
## Local Development Setup
15+
16+
This project uses Python 3.11, Docker, and Docker Compose.
17+
18+
This document describes how to set up a development environment using Nix, which is a package manager that allows for reproducible builds and development environments, like a better encapsulated declarative cross-platform Homebrew.
19+
20+
For a basic rundown on Nix, this video could be useful https://www.youtube.com/watch?v=yQwW8dkuHqw
21+
22+
1. Install the following according to the instructions for your platform if not already installed:
23+
1. Direnv - https://direnv.net/docs/installation.html
24+
2. Docker Engine
25+
* Linux - https://docs.docker.com/engine/install/
26+
* MacOS - https://orbstack.dev/ or https://github.com/abiosoft/colima ?
27+
* Windows - ?
28+
3. Just - https://just.systems/man/en/packages.html
29+
4. Nix - https://nixos.org/download/
30+
2. Clone this repository to your machine.
31+
3. In a terminal run `just bootstrap` in the root of the checked out repository to install the necessary development dependencies and generate the .env file.
32+
4. Update the generated .env file with the necessary environment variables. Where you can't retrieve these yourself, you can ask someone for some in #boost-website on the slack server at https://ccplang.slack.com. The minimum that must be set is:
33+
* GITHUB_TOKEN - a personal access token for the GitHub API, from your profile
34+
* STATIC_CONTENT_AWS_ACCESS_KEY_ID - ask for this
35+
* STATIC_CONTENT_AWS_SECRET_ACCESS_KEY - ask for this
36+
* STATIC_CONTENT_BUCKET_NAME - ask for this
37+
* STATIC_CONTENT_REGION - ask for this
38+
* STATIC_CONTENT_AWS_S3_ENDPOINT_URL - ask for this
39+
5. Run `just setup` to build services, and build the JS and CSS assets.
40+
6. Run `docker compose up` to start the server.
41+
* This will start the process of downloading the Boost Version data etc.
42+
43+
### Social Login with django-allauth
44+
45+
Follow these instructions to use the social logins through django-allauth on your local machine.
46+
47+
See https://testdriven.io/blog/django-social-auth/ for more information.
48+
49+
#### Github
50+
- Go to https://github.com/settings/applications/new and add a new OAuth application
51+
- Set `http://localhost:8000` as the Homepage URL
52+
- Set `http://localhost:8000/accounts/github/login/callback/` as the Callback URL
53+
- Click whether you want to enable the device flow
54+
- <img src="https://user-images.githubusercontent.com/2286304/252841283-9a846c68-46bb-4dac-8d1e-d35270c09f1b.png" alt="The GitHub screen that registers a new OAuth app" width="400">
55+
- On completion copy the Client ID and Client Secret to the `.env` file as values of `GITHUB_OAUTH_CLIENT_ID` and `GITHUB_OAUTH_CLIENT_SECRET`.
56+
- Run `direnv allow` and restart your docker containers.
57+
58+
Setup should be complete and you should be able to see an option to "Use Github" on the sign up page.
59+
60+
To test the flow including authorizing Github for the Boost account, log into your GitHub account settings and click **Applications** in the left menu. Find the "Boost" authorization and delete it. The next time you log into Boost with this GitHub account, you will have to re-authorize it.
61+
62+
<img src="https://user-images.githubusercontent.com/2286304/204642346-8b269aaf-4693-4351-9474-0a998b97689c.png" alt="The 'Authorized OAuth Apps' tab in your GitHub Applications" width="400">
63+
64+
This setup process is not something that can currently be automated through terraform because of a lack of relevant Github API endpoints to create Oauth credentials.
65+
66+
#### Google
67+
68+
More detailed instructions at:
69+
70+
https://docs.allauth.org/en/latest/socialaccount/providers/google.html
71+
72+
1. Update the `.env` file with values for:
73+
1. `TF_VAR_google_cloud_email` (the email address of your Google Cloud account)
74+
2. `TF_VAR_google_organization_domain` (usually the domain of your Google Cloud account, e.g. "boost.org" if you will be using a @boost.org email address)
75+
3. `TF_VAR_google_cloud_project_name` (optional, default: localboostdev) - needs to change if destroyed and a setup is needed within 30 days
76+
2. Run `just development-tofu-init` to initialize tofu.
77+
3. Run `just development-tofu-plan` to confirm the planned changes.
78+
4. Run `just development-tofu-apply` to apply the changes.
79+
5. Go to https://console.developers.google.com/
80+
1. Search for the newly created project, named "Boost Development" (ID: localboostdev by default).
81+
2. Type "credentials" in the search input at the top of the page.
82+
3. Select "Credentials" under "APIs & Services".
83+
1. Click "+ CREATE CREDENTIALS"
84+
2. Select "OAuth Client ID"
85+
3. Select Application Type: "Web application"
86+
4. Name: "Boost Development" (arbitrary)
87+
5. For "Authorized Javascript Origins" use:`http://localhost:8000`
88+
6. For "Authorized Redirect URIs" use:
89+
* `http://localhost:8000/accounts/google/login/callback/`
90+
* `http://localhost:8000/accounts/google/login/callback/?flowName=GeneralOAuthFlow`
91+
7. Save
92+
6. From the page that's displayed, update the `.env` file with values for the following:
93+
- `GOOGLE_OAUTH_CLIENT_ID` should be similar to "k235bn2b1l1(...)asdsk.apps.googleusercontent.com"
94+
- `GOOGLE_OAUTH_CLIENT_SECRET` should be similar to "LAJACO(...)KLAI612ANAD"
95+
7. Run `docker compose down && docker compose up` and restart your docker containers.
96+
97+
Point 5 above can not be automated through terraform because of a lack of relevant Google Cloud API endpoints to create Oauth credentials.
98+
99+
Setup should be complete and you should be able to see an option to "Use Google" on the sign up page.
100+
101+
#### Additional Notes on allauth login flows:
102+
**Working locally**: If you need to run through the login flows multiple times, create a superuser so you can log into the admin. Then, log into the admin and delete your "Social Account" from the admin. This will test a fresh connection to GitHub for your logged-in GitHub user.
103+
104+
## Syncing EmailData Locally (optional)
105+
106+
To work with mailinglist data locally, the django application expects to be
107+
able to query a copy of the hyperkitty database from HYPERKITTY_DATABASE_NAME.
108+
Then, `just manage sync_mailinglist_stats` management command can be run.
109+
110+
## Debugging
111+
For local development there is Django Debug Toolbar, and the option to set a debugger.
112+
113+
In your env:
114+
- Django Debug Toolbar, enabled by default, can be disabled by setting DEBUG_TOOLBAR=False
115+
- IDE Debugging, disabled by default, can be enabled by uncommenting `PYTHONBREAKPOINT` in your .env file.
116+
117+
### Set Up Pycharm
118+
You can set up your IDE with a new "Python Debug Server" configuration as:
119+
120+
<img src="images/pycharm_debugger_settings.png" alt="PyCharm Debugger Settings" width="400">
121+
122+
### Debugger Usage
123+
To use the debugger add `breakpoint()` on a line in the code before you want to start debugging and then add breakpoints by clicking on the gutter. The debugger will stop at these point, you can then step/inspect the variables.
124+
125+
126+
## Troubleshooting
127+
128+
### Docker
129+
Keep in mind if there are issues with docker that the host docker daemon on your machine and the docker daemon in the nix setup may not match. It's a good idea to keep both up to date.
130+
131+
### Direnv
132+
when you switch to the directory you may see direnv exporting a bunch of environment variables as below.
133+
134+
The installer configures direnv to suppress those but it's a recent configuration option, so may be worth checking for an update if you see them.
135+
136+
## Disk space
137+
Should you find you're running short on disk space, to delete previous versioned store data you can run `nix-collect-garbage -d`. Reentering the directory will then reinstall all the current dependencies again. It's probably a good idea to run that periodically.
138+
139+
```shell
140+
direnv: export +ALLOWED_HOSTS +AR +AS...
141+
```

‎flake.lock

Lines changed: 61 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎flake.nix

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
{
2+
description = "Boost.org development environment.";
3+
4+
inputs = {
5+
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
6+
flake-utils.url = "github:numtide/flake-utils";
7+
};
8+
9+
outputs = { self, nixpkgs, flake-utils, ... }@inputs:
10+
flake-utils.lib.eachDefaultSystem (system:
11+
let
12+
pkgs = import nixpkgs {
13+
inherit system;
14+
};
15+
# https://nixos.wiki/wiki/Google_Cloud_SDK
16+
gdk = pkgs.google-cloud-sdk.withExtraComponents( with pkgs.google-cloud-sdk.components; [
17+
gke-gcloud-auth-plugin
18+
]);
19+
# Install a Ruby gem from rubygems.org
20+
asciidoctorBoostGem = pkgs.stdenv.mkDerivation rec {
21+
pname = "asciidoctor-boost";
22+
version = "0.1.7";
23+
sha = "ce139448812a9848219ce4cdb521c83c16009406a9d16efbc90bb24e94a46c24";
24+
25+
src = pkgs.fetchurl {
26+
url = "https://rubygems.org/downloads/${pname}-${version}.gem";
27+
sha256 = "${sha}";
28+
};
29+
dontUnpack = true;
30+
nativeBuildInputs = [ pkgs.ruby ];
31+
buildPhase = "true"; # Nothing to compile.
32+
installPhase = ''
33+
# Create a temporary gem directory
34+
mkdir -p $out
35+
# Set GEM_HOME to install gems locally under $out.
36+
export GEM_HOME=$out
37+
# Install the gem into GEM_HOME.
38+
${pkgs.ruby}/bin/gem install ${src} --no-document --ignore-dependencies
39+
'';
40+
meta = {
41+
description = "Asciidoctor Boost Ruby Gem installed from rubygems.org";
42+
homepage = "https://rubygems.org/gems/asciidoctor-boost";
43+
license = "BSL-1.0";
44+
};
45+
};
46+
47+
in {
48+
devShells.default = pkgs.mkShell {
49+
buildInputs = with pkgs; [
50+
# general system
51+
# e.g. this could contain docker client if we wanted that to be consistent,
52+
# though we need the daemon on the host anyway so it's redundant
53+
# general project
54+
gdk
55+
just
56+
opentofu
57+
# frontend
58+
nodejs_22 # matches Dockerfile, due for upgrade?
59+
yarn
60+
# backend
61+
asciidoctor
62+
asciidoctorBoostGem
63+
pre-commit
64+
python311 # matches Dockerfile, due for upgrade?
65+
python311.pkgs.black
66+
python311.pkgs.isort
67+
python311.pkgs.pip-tools
68+
];
69+
# Host system installation workflow goes into the bootstrap justfile target.
70+
# Project specific installation and execution workflow should go here.
71+
shellHook = ''
72+
if [ ! -f .git/hooks/pre-commit ]; then
73+
pre-commit install
74+
fi
75+
if [ ! -d .venv ]; then
76+
python3.11 -m venv .venv
77+
. .venv/bin/activate
78+
pip install -r requirements.txt -r requirements-dev.txt
79+
else
80+
. .venv/bin/activate
81+
fi
82+
if [ ! -f .env ]; then
83+
cp env.template .env
84+
echo ".env created, you should update its contents"
85+
fi
86+
'';
87+
};
88+
}
89+
);
90+
}

‎justfile

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,20 @@ ENV_FILE := ".env"
1212
# ----
1313

1414
@bootstrap: ## installs/updates all dependencies
15-
#!/usr/bin/env bash
16-
set -euo pipefail
17-
if [ ! -f "{{ENV_FILE}}" ]; then
18-
echo "{{ENV_FILE}} created"
19-
cp env.template {{ENV_FILE}}
15+
command -v direnv >/dev/null 2>&1 || { echo >&2 "Direnv is required but not installed. see: https://direnv.net/docs/installation.html - Aborting."; exit 1; }
16+
command -v nix >/dev/null 2>&1 || { echo >&2 "Nix is required but not installed. see: https://nixos.org/download.html - Aborting."; exit 1; }
17+
command -v just >/dev/null 2>&1 || { echo >&2 "Just is required but not installed. see: https://just.systems/man/en/packages.html - Aborting."; exit 1; }
18+
command -v docker >/dev/null 2>&1 || { echo >&2 "Docker is required but not installed. see: docs for links - Aborting."; exit 1; }
19+
if [ ! -d $HOME/.config/direnv/direnv.toml ]; then \
20+
mkdir -p $HOME/.config/direnv; \
21+
printf "[global]\nhide_env_diff = true\nload_dotenv = true\n" > $HOME/.config/direnv/direnv.toml; \
2022
fi
21-
docker compose --file {{COMPOSE_FILE}} build --force-rm
23+
if [ ! -d $HOME/.config/nix ]; then \
24+
mkdir -p $HOME/.config/nix; \
25+
printf "experimental-features = nix-command flakes\n" > $HOME/.config/nix/nix.conf; \
26+
fi
27+
echo "Bootstrapping complete, update your .env and run 'just setup'"
28+
2229

2330
@rebuild: ## rebuilds containers
2431
docker compose kill
@@ -31,7 +38,7 @@ ENV_FILE := ".env"
3138

3239
@build: ## builds containers
3340
docker compose pull
34-
DOCKER_BUILDKIT=1 docker compose build
41+
docker compose build
3542

3643
@cibuild: ## invoked by continuous integration servers to run tests
3744
python -m pytest
@@ -49,6 +56,7 @@ alias shell := console
4956
@setup: ## sets up a project to be used for the first time
5057
docker compose --file {{COMPOSE_FILE}} build --force-rm
5158
docker compose --file docker-compose.yml run --rm web python manage.py migrate --noinput
59+
npm run build
5260

5361
@test_pytest: ## runs pytest
5462
-docker compose run --rm -e DEBUG_TOOLBAR="False" web pytest -s --create-db

0 commit comments

Comments
 (0)