Cubed Lithops Runtime Builder Template

Author: tomwhite

.github/workflows/build-runtime.yml

@@ -0,0 +1,82 @@
+name: Build and Deploy Lithops Runtime
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'Dockerfile'
+      - '.lithops/config'
+  workflow_dispatch:
+
+permissions:
+  id-token: write
+  contents: read
+
+env:
+  PYTHON_VERSION: '3.12'
+  RUNTIME_NAME: 'cubed-runtime'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Read AWS region from config
+        id: config
+        run: |
+          region=$(python3 -c "
+          import yaml
+          with open('.lithops/config') as f:
+              c = yaml.safe_load(f)
+          print(c['aws']['region'].strip())
+          ")
+          echo "region=$region" >> $GITHUB_OUTPUT
+
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
+          aws-region: ${{ steps.config.outputs.region }}
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Install Lithops
+        run: pip install 'lithops[aws]'
+
+      - name: Copy Lithops config
+        run: |
+          mkdir -p ~/.lithops
+          grep -v 'log_filename' .lithops/config > ~/.lithops/config
+
+      - name: Delete existing runtime
+        run: lithops runtime delete -d ${{ env.RUNTIME_NAME }} -b aws_lambda
+
+      - name: Build runtime
+        run: lithops runtime build ${{ env.RUNTIME_NAME }} -b aws_lambda -f Dockerfile
+
+      - name: Grant Lambda access to ECR repository
+        run: |
+          set -euo pipefail
+          runtime="${{ env.RUNTIME_NAME }}"
+          repo=$(aws ecr describe-repositories \
+            --query 'repositories[].repositoryName' \
+            --output text | tr '\t' '\n' | grep "$runtime" | head -1)
+          echo "Setting ECR policy on: $repo"
+          aws ecr set-repository-policy \
+            --repository-name "$repo" \
+            --policy-text '{
+              "Version": "2012-10-17",
+              "Statement": [{
+                "Sid": "LambdaECRAccess",
+                "Effect": "Allow",
+                "Principal": {"Service": "lambda.amazonaws.com"},
+                "Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer"]
+              }]
+            }'
+
+      - name: Deploy runtime
+        run: lithops runtime deploy ${{ env.RUNTIME_NAME }} -b aws_lambda

.lithops/config

@@ -0,0 +1,14 @@
+lithops:
+    backend: aws_lambda
+    storage: aws_s3
+    log_filename: lithops.log  # comment out to see logs in the console
+    # log_level: WARNING
+
+aws:
+    region: us-east-1                                  # your AWS region
+
+aws_lambda:
+    execution_role: arn:aws:iam::...                   # LambdaExecutionRoleArn
+    user_id: AROAXXXXXXXXXXXXXXXXX                     # GitHubActionsRoleId
+    runtime_memory: 2000
+    runtime_timeout: 180

Dockerfile

@@ -0,0 +1,58 @@
+# Python 3.12
+FROM python:3.12-slim-bookworm
+
+RUN apt-get update \
+    # Install aws-lambda-cpp build dependencies
+    && apt-get install -y \
+      g++ \
+      make \
+      cmake \
+      unzip \
+      git \
+    # cleanup package lists, they are not used anymore in this image
+    && rm -rf /var/lib/apt/lists/* \
+    && apt-cache search linux-headers-generic
+
+ARG FUNCTION_DIR="/function"
+
+# Copy function code
+RUN mkdir -p ${FUNCTION_DIR}
+
+# Update pip
+RUN pip install --upgrade --ignore-installed pip wheel six setuptools \
+    && pip install --upgrade --no-cache-dir --ignore-installed \
+        awslambdaric \
+        botocore \
+        boto3 \
+        redis \
+        httplib2 \
+        requests \
+        numpy \
+        scipy \
+        pandas \
+        pika \
+        kafka-python \
+        cloudpickle \
+        ps-mem \
+        tblib
+
+# Set working directory to function root directory
+WORKDIR ${FUNCTION_DIR}
+
+# Add Lithops
+COPY lithops_lambda.zip ${FUNCTION_DIR}
+RUN unzip lithops_lambda.zip \
+    && rm lithops_lambda.zip \
+    && mkdir handler \
+    && touch handler/__init__.py \
+    && mv entry_point.py handler/
+
+RUN pip install \
+        cubed \
+        obstore
+
+# Add extra dependencies here
+# RUN pip install my-package another-package
+
+ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
+CMD [ "handler.entry_point.lambda_handler" ]

README.md

@@ -0,0 +1,92 @@
+# Cubed Lithops Runtime Builder
+
+A GitHub template repository for building and deploying [Lithops](https://lithops-cloud.github.io/) Lambda runtimes for [Cubed](https://github.com/cubed-dev/cubed) via CI — no local Docker required.
+
+When you push changes to the `Dockerfile`, GitHub Actions builds a Docker image and deploys it as a Lambda container runtime named `cubed-runtime`.
+
+## Prerequisites
+
+- An AWS account
+- A GitHub account
+- The [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
+
+## Setup
+
+### 1. Create this repo from the template
+
+Click **Use this template** → **Create a new repository**.
+
+### 2. Bootstrap AWS
+
+Run this once from your local machine. Replace `YOUR_ORG` with your GitHub username or organisation (e.g. `octocat`) and `YOUR_REPO` with the name you gave this repository.
+
+```bash
+aws cloudformation deploy \
+  --template-file cloudformation/github-oidc-role.yml \
+  --stack-name cubed-lithops-github-actions \
+  --parameter-overrides GitHubOrg=YOUR_ORG GitHubRepo=YOUR_REPO \
+  --capabilities CAPABILITY_NAMED_IAM
+```
+
+> **Already have a GitHub OIDC provider?** Use `--parameter-overrides GitHubOrg=YOUR_ORG GitHubRepo=YOUR_REPO CreateOIDCProvider=false` to skip creating one.
+
+This creates a GitHub OIDC identity provider (if one doesn't already exist), an IAM role for GitHub Actions, and an IAM role for Lambda to assume when running functions. When it completes, retrieve both ARNs:
+
+```bash
+aws cloudformation describe-stacks \
+  --stack-name cubed-lithops-github-actions \
+  --query 'Stacks[0].Outputs' \
+  --output table
+```
+
+### 3. Add the secret
+
+In your repo: **Settings → Secrets and variables → Actions → New repository secret**
+
+| Name | Value |
+|------|-------|
+| `AWS_ROLE_ARN` | `GitHubActionsRoleArn` from the previous step |
+
+### 4. Edit `.lithops/config`
+
+Replace the placeholder values:
+
+```yaml
+aws:
+    region: us-east-1                                         # your AWS region
+
+aws_lambda:
+    execution_role: arn:aws:iam::...   # LambdaExecutionRoleArn from the previous step
+    user_id: AROAXXXXXXXXXXXXXXXXX     # GitHubActionsRoleId from the previous step
+```
+
+### 5. Add your dependencies
+
+Edit the `Dockerfile` to add extra packages (there is a clearly marked section near the bottom), then push — the CI pipeline builds and deploys the `cubed-runtime` Lambda runtime automatically.
+
+## Manual trigger
+
+You can also trigger a build from **Actions → Build and Deploy Lithops Runtime → Run workflow**.
+
+## Customisation
+
+### Changing the runtime name
+
+The runtime is named `cubed-runtime` by default. To use a different name, edit the `RUNTIME_NAME` env var at the top of `.github/workflows/build-runtime.yml`:
+
+```yaml
+env:
+  RUNTIME_NAME: 'my-runtime-name'
+```
+
+Remember to also update the `runtime` key in your local Lithops config if you change this.
+
+### Using the latest Cubed from GitHub
+
+To use the latest development version of Cubed from the `main` branch instead of the PyPI release, replace the `cubed` line in the `Dockerfile`:
+
+```dockerfile
+RUN pip install \
+        'git+https://github.com/cubed-dev/cubed.git#egg=cubed' \
+        obstore
+```