Refresh of Jupyter environment

Author: cbanner

.gitignore

@@ -2,14 +2,10 @@
 .env
 .pyc
 venv
+.venv
 __pycache__
 build/
 
-# Anaconda
-envs
-.conda
-.cache
-
 # OS X Artifacts
 .DS_Store
 

CLAUDE.md

@@ -0,0 +1,119 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Overview
+
+This repository contains Optimizely Labs - a collection of self-contained tutorials demonstrating how to work with Optimizely data and developer tools. Each lab is published to optimizely.com/labs via a Contentful CMS integration.
+
+## Repository Architecture
+
+### Content Structure
+- **`/labs/`**: Each subdirectory is a self-contained lab with its own README.md (tutorial content), metadata.md (Contentful metadata), and any supporting code/resources
+- **`/utils/`**: Python scripts for publishing and managing labs in Contentful CMS
+- **`/templates/`**: Templates for creating new labs (new-content and link-to-existing-content)
+- **`.github/workflows/`**: CI/CD pipelines for automatic publishing to Contentful
+
+### Lab Publication System
+The repository uses a Python-based publishing system that:
+1. Scans `/labs/` for all lab directories (slugs)
+2. Zips each lab's contents and uploads to S3
+3. Parses `metadata.md` (YAML frontmatter) and README.md/index.md
+4. Upserts lab entries to Contentful CMS
+5. Labs appear at `optimizely.com/labs/{slug}`
+
+**Key files:**
+- `utils/publish.py`: Main publishing script - zips labs, uploads to S3, syncs to Contentful
+- `utils/delete.py`: Deletes a specific lab from Contentful
+- `utils/labs_constants.py`: Configuration constants (paths, Contentful settings, S3 credentials)
+
+### Lab Slug Requirements
+- Slugs must match `^[a-zA-Z0-9-.]{1,64}$` (alphanumeric, dashes, periods, underscores only)
+- Slugs serve as both directory names and Contentful entry IDs
+- Slugs become URL paths: `optimizely.com/labs/{slug}`
+
+## Common Development Tasks
+
+### Installing Dependencies
+```bash
+pip install -r requirements.txt
+```
+
+### Publishing Labs to Contentful
+Publishing happens automatically on push to `master` branch via GitHub Actions, but can be run manually:
+
+```bash
+# Set required environment variables (see labs_constants.py for full list)
+export LABS_CONTENTFUL_ENVIRONMENT=master
+export LABS_CONTENTFUL_SPACE_ID=zw48pl1isxmc
+export LABS_CONTENTFUL_MANAGEMENT_API_TOKEN=<token>
+export LIBRARY_URL=https://library.optimizely.com/
+export LIBRARY_S3_BUCKET=library-optimizely-com
+export LIBRARY_S3_ACCESS_KEY=<key>
+export LIBRARY_S3_SECRET_KEY=<secret>
+
+# Run publish script
+python utils/publish.py
+```
+
+### Deleting a Lab
+```bash
+export SLUG_TO_DELETE=<lab-slug>
+export LABS_CONTENTFUL_ENVIRONMENT=master
+export LABS_CONTENTFUL_SPACE_ID=zw48pl1isxmc
+export LABS_CONTENTFUL_MANAGEMENT_API_TOKEN=<token>
+
+python utils/delete.py
+```
+
+### Creating a New Lab
+
+1. Create directory under `/labs/` with a valid slug name
+2. Add `README.md` using template from `/templates/new-content/README.md` or `/templates/link-to-existing-content/README.md`
+3. Add `metadata.md` using template from `/templates/metadata.md`
+4. Set `excludeFromListing: true` in metadata.md initially
+5. Use absolute URLs for links/images (prefix with `https://raw.githubusercontent.com/optimizely/labs/master/` for images)
+6. Submit PR for review
+7. After merge, lab appears at `optimizely.com/labs/{slug}`
+8. When ready to feature, set `excludeFromListing: false` via separate PR
+
+### Lab Content Structure
+
+**README.md/index.md** (content priority: index.md > README.md):
+- Tutorial content in Markdown
+- Should include: summary, pre-requisites, numbered steps
+- Use absolute URLs for all links and images
+
+**metadata.md**:
+- YAML frontmatter with: title, summary, revisionDate, labels, author, seo, excludeFromListing
+- Controls how lab appears in Contentful and on optimizely.com/labs
+
+## Lab Types
+
+Labs cover various Optimizely integrations and use cases:
+- **Feature Flags**: Python Flask, Ruby Sinatra examples
+- **Data Analysis**: Spark/PySpark notebooks for Enriched Event data
+- **Integrations**: Third-party integrations (ContentSquare, Crazy Egg, Zuko, Segment)
+- **Browser Extensions**: Custom Optimizely editor extensions
+- **Event Targeting**: Sequential events, custom targeting examples
+
+### Data Analysis Labs
+Some labs (computing-experiment-metrics, query-enriched-event-data-with-spark) use:
+- Docker for containerized execution
+- PySpark for processing Optimizely Enriched Event data
+- Jupyter Lab for interactive notebooks
+- Python requirements.txt for lab-specific dependencies
+
+Run these labs using:
+```bash
+bash bin/run.sh           # Uses Docker (primary method)
+bash bin/run-docker.sh    # Direct Docker execution
+```
+
+## Important Notes
+
+- Labs are self-contained - all resources should be within the lab directory
+- Publishing is automatic on master branch commits
+- Lab zips are uploaded to S3 and linked in Contentful as downloadable resources
+- Metadata field `excludeFromListing: true` hides labs from main listing page
+- Use GitHub Actions workflow "Contentful Delete" to manually delete labs

labs/computing-experiment-metrics/Dockerfile

@@ -0,0 +1,33 @@
+# Use official Python image
+FROM python:3.11-slim
+
+# Install Java (required for PySpark)
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+    openjdk-21-jre-headless \
+    wget \
+    curl \
+    && apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Set Java environment variables
+ENV JAVA_HOME=/usr/lib/jvm/java-21-openjdk-arm64
+
+# Install PySpark and Jupyter via pip
+RUN pip install --no-cache-dir \
+    pyspark==3.5.0 \
+    jupyterlab==4.0.9 \
+    notebook \
+    ipykernel \
+    ipywidgets
+
+# Create a non-root user
+RUN useradd -m -s /bin/bash jovyan
+USER jovyan
+WORKDIR /home/jovyan
+
+# Expose Jupyter port
+EXPOSE 8888
+
+# Default command
+CMD ["jupyter", "lab", "--ip=0.0.0.0", "--port=8888", "--no-browser"]

labs/computing-experiment-metrics/README.md

@@ -4,78 +4,36 @@ In this Lab, we'll walk through an end-to-end workflow for computing a series of
 
 ## Running this notebook
 
-There are several ways to run this notebook locally:
-- Using the `run.sh` script
-- Using [Docker](https://www.docker.com/) with the `run-docker.sh` script
-- Manually, using the `conda` CLI
+This lab runs using [Docker](https://www.docker.com/), which provides all necessary dependencies (Python, Java, Spark, Jupyter). The lab includes a `Dockerfile` that builds a custom image from the official `python:3.11-slim` base image, installing all dependencies via standard package managers (apt and pip).
 
-### Running the notebook with `run.sh`
-
-You can use the `run.sh` script to build your environment and run this notebook with a single command.
-
-#### Prerequisite: conda (version 4.4+)
-
-[Anaconda]: https://www.anaconda.com/distribution/
-[Miniconda]: https://docs.conda.io/en/latest/miniconda.html
+### Prerequisites
 
-You can install the `conda` CLI by installing [Anaconda] or [Miniconda].
+- [Docker](https://www.docker.com/) must be installed on your system
 
-#### Running Jupyter Lab
+### Running the notebook with `run.sh`
 
-This lab directory contains a handy script for building your conda environment and running Jupyter Lab.  To run it, simply use
+You can use the `run.sh` script to run this notebook with a single command:
 
 ```sh
 bash bin/run.sh
 ```
 
-That's it, you're done!
+This script will:
+1. Build a custom Docker image from the local Dockerfile
+2. Install lab-specific Python dependencies
+3. Launch Jupyter Lab
 
-### Running this notebook with Docker
+That's it, you're done!
 
-If you have [Docker](https://www.docker.com/) installed, you can run PySpark and Jupyter Lab without installing any other dependencies.
+### Running the notebook directly with Docker
 
-Execute `run-docker.sh` in the `./bin` directory to open Jupyter Lab in a Docker container:
+Alternatively, you can execute `run-docker.sh` directly:
 
 ```sh
 bash bin/run-docker.sh
 ```
 
-**Note:** Docker makes it easy to get started with PySpark, but it adds overhead and may require [additional configuration](https://docs.docker.com/config/containers/resource_constraints/) to handle large workloads.  
-
-### Running this notebook manually
-
-If you prefer to build and activate your conda environment manually, you can use the `conda` CLI and the environment specification files in the `./lab_env` directory to do so.
-
-#### Prerequisite: conda (version 4.4+)
-
-[Anaconda]: https://www.anaconda.com/distribution/
-[Miniconda]: https://docs.conda.io/en/latest/miniconda.html
-
-You can install the `conda` CLI by installing [Anaconda] or [Miniconda].
-
-#### Building and activating your Aanconda environment
-
-Start by building (or updating) and activating your anaconda environment.  This step will install [OpenJDK](https://openjdk.java.net/), [PySpark](https://spark.apache.org/docs/latest/api/python/pyspark.html), [Jupyter Lab](https://jupyter.org/), and other necessary dependencies.
-
-```sh
-conda env update --file lab_env/base.yml --name optimizelylabs
-conda env update --file lab_env/labs.yml --name optimizelylabs
-conda activate optimizelylabs
-```
-
-Next, install a jupyter kernel for this environment:
-
-```sh
-python -m ipykernel install --user \
-    --name optimizelylabs \
-    --display-name="Python 3 (Optimizely Labs Environment)"
-```
-
-Finally, start Jupyter Lab in your working directory:
-
-```sh
-jupyter lab .
-```
+**Note:** Docker makes it easy to get started with PySpark, but it may require [additional configuration](https://docs.docker.com/config/containers/resource_constraints/) to handle large workloads.
 
 ## Specifying a custom data directory
 

labs/computing-experiment-metrics/bin/build.sh

@@ -27,13 +27,20 @@ if [[ ! -f "$NB" ]]; then
     exit 1
 fi
 
-# Create conda build environment
-export INSTALL_BUILD_DEPENDENCIES=true
-. "$LAB_BIN_DIR/env.sh"
+# Create build directory
+mkdir -p "$LAB_BUILD_DIR"
+
+# Create Python virtual environment and install build dependencies
+echo "Creating Python virtual environment for build"
+VENV_DIR="$LAB_BUILD_DIR/.venv"
+python3 -m venv "$VENV_DIR"
+source "$VENV_DIR/bin/activate"
+
+echo "Installing build dependencies"
+pip install -r "$LAB_BASE_DIR/requirements-build.txt"
 
 # Backup passed notebook
 echo "Backing up $NB to $LAB_BUILD_DIR/backup.ipynb"
-mkdir -p "$LAB_BUILD_DIR"
 cp "$NB" "$LAB_BUILD_DIR/backup.ipynb"
 
 # 1. Remove outputs and cell metadata from the passed notebook

labs/computing-experiment-metrics/bin/env.sh

@@ -3,17 +3,32 @@
 # run-docker.sh
 #
 # A handy script for running the lab notebook locally in a docker container
+# Uses a custom Docker image built from python:3.11-slim
 
 set -e
 
 # Use the script path to build an absolute path for the Lab's base directory
 SCRIPT_DIR=$(dirname "$0")
 . "$SCRIPT_DIR/base.sh"
 
+# Build the Docker image from local Dockerfile
+IMAGE_NAME="optimizely-pyspark-lab"
+echo "Building Docker image..."
+docker build -t "$IMAGE_NAME" "$LAB_BASE_DIR"
+
 # The Lab directory should be mounted in ~/lab in the container
 CONTAINER_HOME=/home/jovyan
 CONTAINER_LAB_BASE_DIR="$CONTAINER_HOME/lab"
-CONTAINER_LAB_BIN_DIR="$CONTAINER_LAB_BASE_DIR/bin"
+
+# Build the startup command to install dependencies and launch Jupyter
+STARTUP_CMD="pip install -r $CONTAINER_LAB_BASE_DIR/requirements.txt && \
+jupyter lab --ip=0.0.0.0 --port=8888 --no-browser --NotebookApp.token='' --NotebookApp.password='' --notebook-dir=$CONTAINER_LAB_BASE_DIR"
+
+# Detect if running in interactive terminal
+DOCKER_FLAGS=""
+if [ -t 0 ]; then
+    DOCKER_FLAGS="-it"
+fi
 
 # If OPTIMIZELY_DATA_DIR is defined, mount the specified data directory in
 # the container and set the container OPTIMIZELY_DATA_DIR envar accordingly
@@ -22,21 +37,17 @@ if [[ -n "${OPTIMIZELY_DATA_DIR:-}" ]]; then
     CONTAINER_DATA_DIR="$CONTAINER_HOME/optimizely_data"
     echo "OPTIMIZELY_DATA_DIR envar set.  Mapping to $CONTAINER_DATA_DIR"
 
-    docker run -it --rm \
+    docker run $DOCKER_FLAGS --rm \
         -p 8888:8888 \
         -v "$LAB_BASE_DIR:$CONTAINER_LAB_BASE_DIR" \
         -v "$OPTIMIZELY_DATA_DIR:$CONTAINER_DATA_DIR" \
-        -e "IN_DOCKER_CONTAINER=true" \
         -e "OPTIMIZELY_DATA_DIR=$CONTAINER_DATA_DIR" \
-        jupyter/pyspark-notebook \
-        bash "$CONTAINER_LAB_BIN_DIR/run.sh"
+        "$IMAGE_NAME" \
+        bash -c "$STARTUP_CMD"
 else
-    docker run -it --rm \
+    docker run $DOCKER_FLAGS --rm \
         -p 8888:8888 \
         -v "$LAB_BASE_DIR:$CONTAINER_LAB_BASE_DIR" \
-        -e "IN_DOCKER_CONTAINER=true" \
-        jupyter/pyspark-notebook \
-        bash "$CONTAINER_LAB_BIN_DIR/run.sh"
+        "$IMAGE_NAME" \
+        bash -c "$STARTUP_CMD"
 fi
-
-