Skip to content

Commit 8a6df7d

Browse files
tomwhitetomwhite
committed
Cubed Lithops Runtime Builder Template
0 parents  commit 8a6df7d

5 files changed

Lines changed: 453 additions & 0 deletions

File tree

‎.github/workflows/build-runtime.yml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
name: Build and Deploy Lithops Runtime
2+
3+
on:
4+
push:
5+
branches: [main]
6+
paths:
7+
- 'Dockerfile'
8+
- '.lithops/config'
9+
workflow_dispatch:
10+
11+
permissions:
12+
id-token: write
13+
contents: read
14+
15+
env:
16+
PYTHON_VERSION: '3.12'
17+
RUNTIME_NAME: 'cubed-runtime'
18+
19+
jobs:
20+
check:
21+
runs-on: ubuntu-latest
22+
outputs:
23+
configured: ${{ steps.check.outputs.configured }}
24+
steps:
25+
- id: check
26+
env:
27+
SECRET: ${{ secrets.AWS_ROLE_ARN }}
28+
run: |
29+
if [ -n "$SECRET" ]; then
30+
echo "configured=true" >> $GITHUB_OUTPUT
31+
else
32+
echo "configured=false" >> $GITHUB_OUTPUT
33+
fi
34+
35+
build:
36+
needs: check
37+
if: needs.check.outputs.configured == 'true'
38+
runs-on: ubuntu-latest
39+
steps:
40+
- uses: actions/checkout@v4
41+
42+
- name: Read AWS region from config
43+
id: config
44+
run: |
45+
region=$(python3 -c "
46+
import yaml
47+
with open('.lithops/config') as f:
48+
c = yaml.safe_load(f)
49+
print(c['aws']['region'].strip())
50+
")
51+
echo "region=$region" >> $GITHUB_OUTPUT
52+
53+
- name: Configure AWS credentials
54+
uses: aws-actions/configure-aws-credentials@v4
55+
with:
56+
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
57+
aws-region: ${{ steps.config.outputs.region }}
58+
59+
- name: Set up Python
60+
uses: actions/setup-python@v5
61+
with:
62+
python-version: ${{ env.PYTHON_VERSION }}
63+
64+
- name: Install Lithops
65+
run: pip install 'lithops[aws]'
66+
67+
- name: Copy Lithops config
68+
run: |
69+
mkdir -p ~/.lithops
70+
grep -v 'log_filename' .lithops/config > ~/.lithops/config
71+
72+
- name: Delete existing runtime
73+
run: lithops runtime delete -d ${{ env.RUNTIME_NAME }} -b aws_lambda
74+
75+
- name: Build runtime
76+
run: lithops runtime build ${{ env.RUNTIME_NAME }} -b aws_lambda -f Dockerfile
77+
78+
- name: Grant Lambda access to ECR repository
79+
run: |
80+
set -euo pipefail
81+
region="${{ steps.config.outputs.region }}"
82+
runtime="${{ env.RUNTIME_NAME }}"
83+
policy='{
84+
"Version": "2012-10-17",
85+
"Statement": [{
86+
"Sid": "LambdaECRAccess",
87+
"Effect": "Allow",
88+
"Principal": {"Service": "lambda.amazonaws.com"},
89+
"Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer"]
90+
}]
91+
}'
92+
repos=$(aws ecr describe-repositories \
93+
--region "$region" \
94+
--query 'repositories[].repositoryName' \
95+
--output text | tr '\t' '\n' | grep -E "(^|/)${runtime}$" || true)
96+
if [ -z "$repos" ]; then
97+
echo "ERROR: no ECR repository found for runtime '$runtime' in $region"
98+
exit 1
99+
fi
100+
while IFS= read -r repo; do
101+
echo "Setting ECR policy on: $repo"
102+
aws ecr set-repository-policy \
103+
--region "$region" \
104+
--repository-name "$repo" \
105+
--policy-text "$policy"
106+
done <<< "$repos"
107+
108+
- name: Deploy runtime
109+
run: lithops runtime deploy ${{ env.RUNTIME_NAME }} -b aws_lambda

‎.lithops/config‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
lithops:
2+
backend: aws_lambda
3+
storage: aws_s3
4+
log_filename: lithops.log # comment out to see logs in the console
5+
# log_level: WARNING
6+
7+
aws:
8+
region: us-east-1 # your AWS region
9+
10+
aws_lambda:
11+
execution_role: arn:aws:iam::... # LambdaExecutionRoleArn
12+
user_id: AROAXXXXXXXXXXXXXXXXX # GitHubActionsRoleId
13+
runtime_memory: 2000
14+
runtime_timeout: 180

‎Dockerfile‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Python 3.12
2+
FROM python:3.12-slim-bookworm
3+
4+
RUN apt-get update \
5+
# Install aws-lambda-cpp build dependencies
6+
&& apt-get install -y \
7+
g++ \
8+
make \
9+
cmake \
10+
unzip \
11+
git \
12+
# cleanup package lists, they are not used anymore in this image
13+
&& rm -rf /var/lib/apt/lists/* \
14+
&& apt-cache search linux-headers-generic
15+
16+
ARG FUNCTION_DIR="/function"
17+
18+
# Copy function code
19+
RUN mkdir -p ${FUNCTION_DIR}
20+
21+
# Update pip
22+
RUN pip install --upgrade --ignore-installed pip wheel six setuptools \
23+
&& pip install --upgrade --no-cache-dir --ignore-installed \
24+
awslambdaric \
25+
botocore \
26+
boto3 \
27+
redis \
28+
httplib2 \
29+
requests \
30+
numpy \
31+
scipy \
32+
pandas \
33+
pika \
34+
kafka-python \
35+
cloudpickle \
36+
ps-mem \
37+
tblib
38+
39+
# Set working directory to function root directory
40+
WORKDIR ${FUNCTION_DIR}
41+
42+
# Add Lithops
43+
COPY lithops_lambda.zip ${FUNCTION_DIR}
44+
RUN unzip lithops_lambda.zip \
45+
&& rm lithops_lambda.zip \
46+
&& mkdir handler \
47+
&& touch handler/__init__.py \
48+
&& mv entry_point.py handler/
49+
50+
RUN pip install \
51+
cubed \
52+
obstore
53+
54+
# Add extra dependencies here
55+
# RUN pip install my-package another-package
56+
57+
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
58+
CMD [ "handler.entry_point.lambda_handler" ]

‎README.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Cubed Lithops Runtime Builder
2+
3+
A GitHub template repository for building and deploying [Lithops](https://lithops-cloud.github.io/) Lambda runtimes for [Cubed](https://github.com/cubed-dev/cubed) via CI — no local Docker required.
4+
5+
When you push changes to the `Dockerfile`, GitHub Actions builds a Docker image and deploys it as a Lambda container runtime named `cubed-runtime`.
6+
7+
## Prerequisites
8+
9+
- An AWS account
10+
- A GitHub account
11+
- The [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
12+
13+
## Setup
14+
15+
### 1. Create this repo from the template
16+
17+
Click **Use this template** → **Create a new repository**. You can name it anything you like — `cubed-lithops-runtime-builder` is a good default.
18+
19+
Then clone it to your local machine:
20+
21+
```bash
22+
git clone https://github.com/YOUR_ORG/YOUR_REPO.git
23+
cd YOUR_REPO
24+
```
25+
26+
### 2. Bootstrap AWS
27+
28+
Run this once from your local machine. Replace `YOUR_ORG` with your GitHub username or organisation (e.g. `octocat`) and `YOUR_REPO` with the name you gave this repository.
29+
30+
```bash
31+
aws cloudformation deploy \
32+
--template-file cloudformation/github-oidc-role.yml \
33+
--stack-name cubed-lithops-github-actions \
34+
--parameter-overrides GitHubOrg=YOUR_ORG GitHubRepo=YOUR_REPO \
35+
--capabilities CAPABILITY_NAMED_IAM
36+
```
37+
38+
> **Already have a GitHub OIDC provider?** Use `--parameter-overrides GitHubOrg=YOUR_ORG GitHubRepo=YOUR_REPO CreateOIDCProvider=false` to skip creating one.
39+
40+
This creates a GitHub OIDC identity provider (if one doesn't already exist), an IAM role for GitHub Actions, and an IAM role for Lambda to assume when running functions. When it completes, retrieve both ARNs:
41+
42+
```bash
43+
aws cloudformation describe-stacks \
44+
--stack-name cubed-lithops-github-actions \
45+
--query 'Stacks[0].Outputs' \
46+
--output table \
47+
--no-cli-pager
48+
```
49+
50+
### 3. Add the secret
51+
52+
In your repo: **Settings → Secrets and variables → Actions → New repository secret**
53+
54+
| Name | Value |
55+
|------|-------|
56+
| `AWS_ROLE_ARN` | `GitHubActionsRoleArn` from the previous step |
57+
58+
### 4. Edit `.lithops/config`
59+
60+
Replace the placeholder values:
61+
62+
```yaml
63+
aws:
64+
region: us-east-1 # your AWS region
65+
66+
aws_lambda:
67+
execution_role: arn:aws:iam::... # LambdaExecutionRoleArn from the previous step
68+
user_id: AROAXXXXXXXXXXXXXXXXX # GitHubActionsRoleId from the previous step
69+
```
70+
71+
### 5. Add your dependencies
72+
73+
Edit the `Dockerfile` to add extra packages (there is a clearly marked section near the bottom), then commit and push with git — the CI pipeline builds and deploys the `cubed-runtime` Lambda runtime automatically.
74+
75+
## Security note
76+
77+
The `.lithops/config` you commit contains your AWS account ID (inside the `execution_role` ARN). This is also visible in CI build logs. AWS [does not consider the account ID a secret](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-identifiers.html), but as a precaution you may want to keep your repository private.
78+
79+
## Manual trigger
80+
81+
You can also trigger a build from **Actions → Build and Deploy Lithops Runtime → Run workflow**.
82+
83+
## Customisation
84+
85+
### Changing the runtime name
86+
87+
The runtime is named `cubed-runtime` by default. To use a different name, edit the `RUNTIME_NAME` env var at the top of `.github/workflows/build-runtime.yml`:
88+
89+
```yaml
90+
env:
91+
RUNTIME_NAME: 'my-runtime-name'
92+
```
93+
94+
Remember to also update the `runtime` key in your local Lithops config if you change this.
95+
96+
### Using the latest Cubed from GitHub
97+
98+
To use the latest development version of Cubed from the `main` branch instead of the PyPI release, replace the `cubed` line in the `Dockerfile`:
99+
100+
```dockerfile
101+
RUN pip install \
102+
'git+https://github.com/cubed-dev/cubed.git#egg=cubed' \
103+
obstore
104+
```

0 commit comments

Comments
 (0)