Skip to content

Commit 1383aca

Browse files
cabutlermitcabutlermit
committed
Update Workflows
Why these changes are being introduced: This is the first step in migrating this repository to optionally build for either AMD64 or ARM64 CPU Architecture for deployment in AWS. How this addresses that need: * Update the `Makefile` with the new output from the mitlib-tf-workloads-ecr repository (and then make some further modifications that will be reflected back in the mitlib-tf-workloads-ecr repository soon) * Update the dev-build.yml workflow with the new output from the mitlib-tf-workloads-ecr repository * Update the stage-build.yml workflow with the new output from the mitlib-tf-workloads-ecr repository * Update the prod-promote.yml workflow with the new output from the mitlib-tf-workloads-ecr repository * Update the README with notes about building and deploying in AWS Side effects of this change: None. Since we did not create an `.aws-architecture` file, the builds for AWS will still default to AMD64 as before. Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/IN-1448
1 parent e2107bf commit 1383aca

5 files changed

Lines changed: 184 additions & 49 deletions

File tree

‎.github/workflows/dev-build.yml‎

Lines changed: 42 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
1-
### This is the Terraform-generated dev-build.yml workflow for the marimo-launcher-dev app repository ###
2-
### If this is a Lambda repo, uncomment the FUNCTION line at the end of the document ###
3-
### If the container requires any additional pre-build commands, uncomment and edit ###
4-
### the PREBUILD line at the end of the document. ###
1+
### This is the Terraform-generated dev-build.yml workflow for the ###
2+
### marimo-launcher-dev app repository. ###
3+
### If this is a Lambda repo, uncomment the FUNCTION line at the end of ###
4+
### the document. If the container requires any additional pre-build ###
5+
### commands, uncomment and edit the PREBUILD line at the end of the ###
6+
### document. ###
7+
58
name: Dev Container Build and Deploy
69
on:
710
workflow_dispatch:
@@ -11,14 +14,47 @@ on:
1114
paths-ignore:
1215
- '.github/**'
1316

17+
permissions:
18+
id-token: write
19+
contents: read
20+
1421
jobs:
22+
prep:
23+
name: Prep for Build
24+
runs-on: ubuntu-latest
25+
outputs:
26+
cpuarch: ${{ steps.setarch.outputs.cpuarch }}
27+
steps:
28+
- name: Checkout
29+
uses: actions/checkout@v5
30+
31+
- name: Set CPU Architecture
32+
id: setarch
33+
run: |
34+
echo "### :abacus: Architecture Selection" >> $GITHUB_STEP_SUMMARY
35+
if [[ -f .aws-architecture ]]; then
36+
ARCH=$(cat .aws-architecture)
37+
echo "\`$ARCH\` was read from \`.aws-architecture\` and passed to the deploy job." >> $GITHUB_STEP_SUMMARY
38+
else
39+
ARCH="linux/amd64"
40+
echo "No \`.aws-architecture\` file, so default \`$ARCH\` was passed to the deploy job." >> $GITHUB_STEP_SUMMARY
41+
fi
42+
if [[ "$ARCH" != "linux/arm64" && "$ARCH" != "linux/amd64" ]]; then
43+
echo "$ARCH is INVALID architecture!"
44+
echo "$ARCH is INVALID architecture!" >> $GITHUB_STEP_SUMMARY
45+
exit 1
46+
fi
47+
echo "cpuarch=$ARCH" >> $GITHUB_OUTPUT
48+
1549
deploy:
16-
name: Dev Container Deploy
17-
uses: mitlibraries/.github/.github/workflows/ecr-shared-deploy-dev.yml@main
50+
needs: prep
51+
name: Dev Deploy
52+
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-deploy-dev.yml@main
1853
secrets: inherit
1954
with:
2055
AWS_REGION: "us-east-1"
2156
GHA_ROLE: "marimo-launcher-gha-dev"
2257
ECR: "marimo-launcher-dev"
58+
CPU_ARCH: ${{ needs.prep.outputs.cpuarch }}
2359
# FUNCTION: ""
2460
# PREBUILD:

‎.github/workflows/prod-promote.yml‎

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,57 @@
1-
### This is the Terraform-generated prod-promote.yml workflow for the marimo-launcher-prod repository. ###
2-
### If this is a Lambda repo, uncomment the FUNCTION line at the end of the document. ###
1+
### This is the Terraform-generated prod-promote.yml workflow for the ###
2+
### marimo-launcher-prod repository. ###
3+
### If this is a Lambda repo, uncomment the FUNCTION line at the end of ###
4+
### the document. ###
5+
36
name: Prod Container Promote
47
on:
58
workflow_dispatch:
69
release:
710
types: [published]
811

12+
permissions:
13+
id-token: write
14+
contents: read
15+
916
jobs:
17+
prep:
18+
name: Prep for Promote
19+
runs-on: ubuntu-latest
20+
outputs:
21+
cpuarch: ${{ steps.setarch.outputs.cpuarch }}
22+
steps:
23+
- name: Checkout
24+
uses: actions/checkout@v5
25+
26+
- name: Set CPU Architecture
27+
id: setarch
28+
run: |
29+
echo "### :abacus: Architecture Selection" >> $GITHUB_STEP_SUMMARY
30+
if [[ -f .aws-architecture ]]; then
31+
ARCH=$(cat .aws-architecture)
32+
echo "\`$ARCH\` was read from \`.aws-architecture\` and passed to the deploy job." >> $GITHUB_STEP_SUMMARY
33+
else
34+
ARCH="linux/amd64"
35+
echo "No \`.aws-architecture\` file, so default \`$ARCH\` was passed to the deploy job." >> $GITHUB_STEP_SUMMARY
36+
fi
37+
if [[ "$ARCH" != "linux/arm64" && "$ARCH" != "linux/amd64" ]]; then
38+
echo "$ARCH is INVALID architecture!"
39+
echo "$ARCH is INVALID architecture!" >> $GITHUB_STEP_SUMMARY
40+
exit 1
41+
fi
42+
echo "cpuarch=$ARCH" >> $GITHUB_OUTPUT
43+
1044
deploy:
11-
name: Prod Container Promote
12-
uses: mitlibraries/.github/.github/workflows/ecr-shared-promote-prod.yml@main
45+
needs: prep
46+
name: Deploy
47+
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-promote-prod.yml@main
1348
secrets: inherit
1449
with:
1550
AWS_REGION: "us-east-1"
1651
GHA_ROLE_STAGE: marimo-launcher-gha-stage
1752
GHA_ROLE_PROD: marimo-launcher-gha-prod
1853
ECR_STAGE: "marimo-launcher-stage"
1954
ECR_PROD: "marimo-launcher-prod"
55+
CPU_ARCH: ${{ needs.prep.outputs.cpuarch }}
2056
# FUNCTION: ""
2157

‎.github/workflows/stage-build.yml‎

Lines changed: 42 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
1-
### This is the Terraform-generated dev-build.yml workflow for the marimo-launcher-stage app repository ###
2-
### If this is a Lambda repo, uncomment the FUNCTION line at the end of the document ###
3-
### If the container requires any additional pre-build commands, uncomment and edit ###
4-
### the PREBUILD line at the end of the document. ###
1+
### This is the Terraform-generated stage-build.yml workflow for the ###
2+
### marimo-launcher-stage app repository. ###
3+
### If this is a Lambda repo, uncomment the FUNCTION line at the end of ###
4+
### the document. If the container requires any additional pre-build ###
5+
### commands, uncomment and edit the PREBUILD line at the end of the ###
6+
### document. ###
7+
58
name: Stage Container Build and Deploy
69
on:
710
workflow_dispatch:
@@ -11,14 +14,47 @@ on:
1114
paths-ignore:
1215
- '.github/**'
1316

17+
permissions:
18+
id-token: write
19+
contents: read
20+
1421
jobs:
22+
prep:
23+
name: Prep for Build
24+
runs-on: ubuntu-latest
25+
outputs:
26+
cpuarch: ${{ steps.setarch.outputs.cpuarch }}
27+
steps:
28+
- name: Checkout
29+
uses: actions/checkout@v5
30+
31+
- name: Set CPU Architecture
32+
id: setarch
33+
run: |
34+
echo "### :abacus: Architecture Selection" >> $GITHUB_STEP_SUMMARY
35+
if [[ -f .aws-architecture ]]; then
36+
ARCH=$(cat .aws-architecture)
37+
echo "\`$ARCH\` was read from \`.aws-architecture\` and passed to the deploy job." >> $GITHUB_STEP_SUMMARY
38+
else
39+
ARCH="linux/amd64"
40+
echo "No \`.aws-architecture\` file, so default \`$ARCH\` was passed to the deploy job." >> $GITHUB_STEP_SUMMARY
41+
fi
42+
if [[ "$ARCH" != "linux/arm64" && "$ARCH" != "linux/amd64" ]]; then
43+
echo "$ARCH is INVALID architecture!"
44+
echo "$ARCH is INVALID architecture!" >> $GITHUB_STEP_SUMMARY
45+
exit 1
46+
fi
47+
echo "cpuarch=$ARCH" >> $GITHUB_OUTPUT
48+
1549
deploy:
16-
name: Stage Container Deploy
17-
uses: mitlibraries/.github/.github/workflows/ecr-shared-deploy-stage.yml@main
50+
needs: prep
51+
name: Stage Deploy
52+
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-deploy-stage.yml@main
1853
secrets: inherit
1954
with:
2055
AWS_REGION: "us-east-1"
2156
GHA_ROLE: "marimo-launcher-gha-stage"
2257
ECR: "marimo-launcher-stage"
58+
CPU_ARCH: ${{ needs.prep.outputs.cpuarch }}
2359
# FUNCTION: ""
2460
# PREBUILD:

‎Makefile‎

Lines changed: 53 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,12 @@
11
SHELL=/bin/bash
22
DATETIME:=$(shell date -u +%Y%m%dT%H%M%SZ)
33
### This is the Terraform-generated header for marimo-launcher-dev. ###
4-
ECR_NAME_DEV:=marimo-launcher-dev
5-
ECR_URL_DEV:=222053980223.dkr.ecr.us-east-1.amazonaws.com/marimo-launcher-dev
4+
ECR_NAME_DEV := marimo-launcher-dev
5+
ECR_URL_DEV := 222053980223.dkr.ecr.us-east-1.amazonaws.com/marimo-launcher-dev
6+
CPU_ARCH ?= $(shell cat .aws-architecture 2>/dev/null || echo "linux/amd64")
67
### End of Terraform-generated header ###
78

9+
810
help: # Preview Makefile commands
911
@awk 'BEGIN { FS = ":.*#"; print "Usage: make <target>\n\nTargets:" } \
1012
/^[-_[:alpha:]]+:.?*#/ { printf " %-15s%s\n", $$1, $$2 }' $(MAKEFILE_LIST)
@@ -111,30 +113,54 @@ docker-test-run: # Test local docker container with test fixture notebook
111113
# Terraform
112114
####################################
113115

114-
### Terraform-generated Developer Deploy Commands for Dev environment ###
115-
dist-dev: ## Build docker container (intended for developer-based manual build)
116-
docker build --platform linux/amd64 \
117-
-t $(ECR_URL_DEV):latest \
118-
-t $(ECR_URL_DEV):`git describe --always` \
119-
-t $(ECR_NAME_DEV):latest .
116+
### Terraform-generated Developer Deploy Commands for Dev environment ###
117+
check-arch:
118+
@ARCH_FILE=".aws-architecture"; \
119+
if [[ "$(CPU_ARCH)" != "linux/amd64" && "$(CPU_ARCH)" != "linux/arm64" ]]; then \
120+
echo "Invalid CPU_ARCH: $(CPU_ARCH)"; exit 1; \
121+
fi; \
122+
if [[ -f $$ARCH_FILE ]]; then \
123+
echo "latest-$(shell echo $(CPU_ARCH) | cut -d'/' -f2)" > .arch_tag; \
124+
else \
125+
echo "latest" > .arch_tag; \
126+
fi
127+
128+
dist-dev: check-arch ## Build docker container (intended for developer-based manual build)
129+
@ARCH_TAG=$$(cat .arch_tag); \
130+
docker buildx inspect $(ECR_NAME_DEV) >/dev/null 2>&1 || docker buildx create --name $(ECR_NAME_DEV) --use; \
131+
docker buildx use $(ECR_NAME_DEV); \
132+
docker buildx build --platform $(CPU_ARCH) \
133+
--load \
134+
--tag $(ECR_URL_DEV):$$ARCH_TAG \
135+
--tag $(ECR_URL_DEV):make-$$ARCH_TAG \
136+
--tag $(ECR_URL_DEV):make-$(shell git describe --always) \
137+
--tag $(ECR_NAME_DEV):$$ARCH_TAG \
138+
.
120139

121140
publish-dev: dist-dev ## Build, tag and push (intended for developer-based manual publish)
122-
docker login -u AWS -p $$(aws ecr get-login-password --region us-east-1) $(ECR_URL_DEV)
123-
docker push $(ECR_URL_DEV):latest
124-
docker push $(ECR_URL_DEV):`git describe --always`
125-
126-
### Terraform-generated manual shortcuts for deploying to Stage. This requires ###
127-
### that ECR_NAME_STAGE, ECR_URL_STAGE, and FUNCTION_STAGE environment ###
128-
### variables are set locally by the developer and that the developer has ###
129-
### authenticated to the correct AWS Account. The values for the environment ###
130-
### variables can be found in the stage_build.yml caller workflow. ###
131-
dist-stage: ## Only use in an emergency
132-
docker build --platform linux/amd64 \
133-
-t $(ECR_URL_STAGE):latest \
134-
-t $(ECR_URL_STAGE):`git describe --always` \
135-
-t $(ECR_NAME_STAGE):latest .
136-
137-
publish-stage: ## Only use in an emergency
138-
docker login -u AWS -p $$(aws ecr get-login-password --region us-east-1) $(ECR_URL_STAGE)
139-
docker push $(ECR_URL_STAGE):latest
140-
docker push $(ECR_URL_STAGE):`git describe --always`
141+
@ARCH_TAG=$$(cat .arch_tag); \
142+
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $(ECR_URL_DEV); \
143+
docker push $(ECR_URL_DEV):$$ARCH_TAG; \
144+
docker push $(ECR_URL_DEV):make-$$ARCH_TAG; \
145+
docker push $(ECR_URL_DEV):make-$(shell git describe --always); \
146+
echo "Cleaning up dangling Docker images..."; \
147+
docker image prune -f --filter "dangling=true"
148+
149+
150+
### If this is a Lambda repo, uncomment the two lines below ###
151+
# update-lambda-dev: ## Updates the lambda with whatever is the most recent image in the ecr (intended for developer-based manual update)
152+
# @ARCH_TAG=$$(cat .arch_tag); \
153+
# aws lambda update-function-code \
154+
# --region us-east-1 \
155+
# --function-name $(FUNCTION_DEV) \
156+
# --image-uri $(ECR_URL_DEV):make-$$ARCH_TAG
157+
158+
docker-clean: ## Clean up Docker detritus
159+
@ARCH_TAG=$$(cat .arch_tag); \
160+
echo "Cleaning up Docker leftovers (containers, images, builders)"; \
161+
docker rmi -f $(ECR_URL_DEV):$$ARCH_TAG; \
162+
docker rmi -f $(ECR_URL_DEV):make-$$ARCH_TAG; \
163+
docker rmi -f $(ECR_URL_DEV):make-$(shell git describe --always) || true; \
164+
docker rmi -f $(ECR_NAME_DEV):$$ARCH_TAG || true; \
165+
docker buildx rm $(ECR_NAME_DEV) || true
166+
@rm -rf .arch_tag

‎README.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ More information about structuring notebooks and dependencies below in "Preparin
3434
## Preparing Notebooks
3535

3636
### Notebook Location
37+
3738
This CLI expects two primary things to discover the notebook to launch:
3839

3940
1. The root directory of the notebook project (either mounted or a cloned Github repository)
@@ -84,14 +85,13 @@ There are many ways to create this file, [`uv export` is worth consideration](ht
8485

8586
The `Makefile` command `cli-test-reqs-txt-run` will demonstrate this.
8687

87-
8888
## Environment Variables
8989

9090
### Required
9191

9292
```shell
93-
SENTRY_DSN=### If set to a valid Sentry DSN, enables Sentry exception monitoring. This is not needed for local development.
94-
WORKSPACE=### Set to `dev` for local development, this will be set to `stage` and `prod` in those environments by Terraform.
93+
SENTRY_DSN= ### If set to a valid Sentry DSN, enables Sentry exception monitoring. This is not needed for local development.
94+
WORKSPACE= ### Set to `dev` for local development, this will be set to `stage` and `prod` in those environments by Terraform.
9595
```
9696

9797
### Optional
@@ -102,15 +102,14 @@ Set these if you want to override defaults or pass values via env instead of fla
102102
NOTEBOOK_REPOSITORY= ### repository to clone that contains a notebook and any required assets
103103
NOTEBOOK_REPOSITORY_BRANCH= ### optional branch to checkout on clone
104104
NOTEBOOK_MOUNT= ### either local of Docker context, an accessible root directory that contains notebook(s)
105-
NOTEBOOK_PATH=### Relative path of actual notebook .py file based on cloned repository or mounted directory; defaults to "notebook.py"
105+
NOTEBOOK_PATH= ### Relative path of actual notebook .py file based on cloned repository or mounted directory; defaults to "notebook.py"
106106
NOTEBOOK_REQUIREMENTS= ### filepath to install dependencies from, relative to notebook root; if unset assuming dependencies are inline in notebook
107107

108108
NOTEBOOK_MODE= ### how to launch marimo: "run" to execute, "edit" to open the editor; default "run"
109109
NOTEBOOK_HOST= ### host to bind running notebook to
110110
NOTEBOOK_PORT= ### port to serve running notebook on
111111
```
112112

113-
114113
## CLI Commands
115114

116115
### `launcher`
@@ -159,6 +158,8 @@ Options:
159158
--help Show this message and exit.
160159
```
161160

161+
## Building for AWS
162162

163+
This application is designed to also run as a Fargate container in AWS. To build and deploy the container to AWS ECR in our Development Account, use the `make publish-dev` command (you must be authenticated to AWS on the CLI first). The infrastructure to run the container (e.g., an ECS Task Definition, Cluster, and Service) are already in place in AWS, so you can easily launch the service via the AWS Console or via the Lambda function trigger.
163164

164-
165+
The infrastructure can support a container built on either `amd64` or `arm64` CPU architecture. There should be a file named `.aws-architecture` at the root of this repository that contains the default architecture for any builds for AWS, either `linux/amd64` or `linux/arm64`. IF the file does not exist, the builds from either the `Makefile` command or the automated deployment with GitHub Actions will default to `amd64`.

0 commit comments

Comments
 (0)