Skip to content

Commit bf9dc4c

Browse files
vsochvsoch
authored
Add notebook tutorial (#9)
* add notebook tutorial and ci Signed-off-by: vsoch <[email protected]>
1 parent 3158d2d commit bf9dc4c

File tree

91 files changed

+8715
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+8715
-0
lines changed

‎tutorial/notebook/Dockerfile‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
FROM fluxrm/flux-sched:jammy
2+
3+
# Based off of https://github.com/jupyterhub/zero-to-jupyterhub-k8s/tree/main/images/singleuser-sample
4+
# Local usage
5+
# docker run -p 8888:8888 -v $(pwd):/home/jovyan/work test
6+
7+
USER root
8+
9+
ENV NB_USER=jovyan \
10+
NB_UID=1000 \
11+
HOME=/home/jovyan
12+
13+
RUN adduser \
14+
--disabled-password \
15+
--gecos "Default user" \
16+
--uid ${NB_UID} \
17+
--home ${HOME} \
18+
--force-badname \
19+
${NB_USER}
20+
21+
RUN apt-get update \
22+
# && apt-get upgrade -y \
23+
&& apt-get install -y --no-install-recommends \
24+
gcc-10 \
25+
g++-10 \
26+
ca-certificates \
27+
dnsutils \
28+
iputils-ping \
29+
python3 \
30+
python3-dev \
31+
python3-pip \
32+
python3-venv \
33+
openmpi-bin \
34+
openmpi-common \
35+
libopenmpi-dev \
36+
liblz4-dev \
37+
tini \
38+
# requirement for nbgitpuller
39+
git \
40+
&& rm -rf /var/lib/apt/lists/*
41+
42+
COPY ./requirements_venv.txt ./requirements_venv.txt
43+
RUN python3 -m pip install -r requirements_venv.txt
44+
45+
COPY ./requirements.txt ./requirements.txt
46+
RUN python3 -m pip install -r requirements.txt && \
47+
python3 -m pip install ipython==7.34.0 && \
48+
python3 -m IPython kernel install
49+
50+
# This adds the flux-tree command, which is provided in flux-sched source
51+
# but not installed alongside production flux-core
52+
COPY ./flux-tree/* /usr/libexec/flux/cmd/
53+
RUN chmod +x /usr/libexec/flux/cmd/flux-tree*
54+
55+
# Flux accounting
56+
RUN git clone https://github.com/flux-framework/flux-accounting && \
57+
cd flux-accounting && \
58+
./autogen.sh && \
59+
./configure --prefix=/usr && \
60+
make && make install
61+
62+
RUN apt-get update && apt-get install -y nodejs && apt-get clean && rm -rf /var/lib/apt/lists/*
63+
64+
RUN wget https://nodejs.org/dist/v20.15.0/node-v20.15.0-linux-x64.tar.xz && \
65+
apt-get update && apt-get install -y xz-utils && rm -rf /var/lib/apt/lists/* && \
66+
xz -d -v node-v20.15.0-linux-x64.tar.xz && \
67+
tar -C /usr/local --strip-components=1 -xvf node-v20.15.0-linux-x64.tar
68+
69+
# This customizes the launcher UI
70+
# https://jupyter-app-launcher.readthedocs.io/en/latest/usage.html
71+
RUN python3 -m pip install jupyter_app_launcher && \
72+
python3 -m pip install --upgrade jupyter-server && \
73+
python3 -m pip install jupyter-launcher-shortcuts && \
74+
mkdir -p /usr/local/share/jupyter/lab/jupyter_app_launcher
75+
76+
COPY ./tutorial /home/jovyan/
77+
COPY ./docker/jupyter-launcher.yaml /usr/local/share/jupyter/lab/jupyter_app_launcher/jp_app_launcher.yaml
78+
ENV JUPYTER_APP_LAUNCHER_PATH=/usr/local/share/jupyter/lab/jupyter_app_launcher/
79+
80+
# Give jovyan user permissions to tutorial materials
81+
RUN chmod -R 777 ~/ /home/jovyan
82+
83+
WORKDIR $HOME
84+
COPY ./docker/flux-icon.png $HOME/flux-icon.png
85+
86+
# note that previous examples are added via git volume in config.yaml
87+
ENV SHELL=/usr/bin/bash
88+
ENV FLUX_URI_RESOLVE_LOCAL=t
89+
90+
EXPOSE 8888
91+
ENTRYPOINT ["tini", "--"]
92+
93+
# This is for JupyterHub
94+
COPY ./docker/entrypoint.sh /entrypoint.sh
95+
96+
# This is for a local start
97+
COPY ./docker/start.sh /start.sh
98+
99+
RUN mkdir -p $HOME/.local/share && \
100+
chmod 777 $HOME/.local/share
101+
102+
# Quick setup of flux-accounting (not working due to needing system service)
103+
# RUN flux start /bin/bash -c "nohup flux account create-db && flux account-service & flux account add-bank root 1" && \
104+
# flux start flux account add-bank --parent-bank=root default 1 && \
105+
# flux start flux account add-user --username=jovyan --bank=default && \
106+
# flux start flux jobtap load mf_priority.so && \
107+
# flux start flux account-update-db
108+
109+
USER ${NB_USER}
110+
111+
CMD ["flux", "start", "--test-size=4", "jupyter", "lab"]

‎tutorial/notebook/README.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Flux + Jupyter
2+
3+
This tutorial provides a notebook for learning about Flux! It is based off of the official Flux [Tutorials](https://github.com/flux-framework/Tutorials) but slimmed down for easier build and usage.
4+
5+
- [Local Development or Usage](#local-usage)
6+
7+
Pre-requisite: Excitement to learn about Flux!
8+
9+
## Usage
10+
11+
This entire tutorial runs on your local machine with a single container! You will need to [install Docker](https://docs.docker.com/engine/install/). When you have Docker available, you can build and run the tutorial with:
12+
13+
```bash
14+
docker build -t flux-tutorial .
15+
docker network create jupyterhub
16+
17+
# Here is how to run an entirely contained tutorial (the notebook in the container)
18+
docker run --rm -it --entrypoint /start.sh -v /var/run/docker.sock:/var/run/docker.sock --net jupyterhub --name jupyterhub -p 8888:8888 flux-tutorial
19+
```
20+
21+
If you want to develop the ipynb files, you can bind the tutorials directory:
22+
23+
```bash
24+
docker run --rm -it --entrypoint /start.sh -v $PWD/tutorial:/home/jovyan/flux-tutorial-2024 -v /var/run/docker.sock:/var/run/docker.sock --net jupyterhub --name jupyterhub -p 8888:8888 flux-tutorial
25+
```
26+
27+
And then editing and saving will save to your host. You can also File -> Download if you forget to do
28+
this bind. Either way, when the container is running you can open the localhost or 127.0.0.1 (home sweet home!) link in your browser on port 8888. You'll want to go to flux-tutorial-2024 -> notebook to see the notebook.
29+
You'll need to select http only (and bypass the no certificate warning).

‎tutorial/notebook/docker/entrypoint.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/bin/bash
2+
3+
/usr/bin/flux start --test-size=4 /usr/local/bin/jupyterhub-singleuser

‎tutorial/notebook/docker/flux-icon.png‎

115 KB
Loading

‎tutorial/notebook/docker/jupyter-launcher.yaml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
- title: Flux Tutorial Notebook
2+
description: This is the main Flux Framework Tutorial
3+
type: jupyterlab-commands
4+
icon: ./flux-icon.png
5+
source:
6+
- label: Flux Tutorial
7+
id: 'filebrowser:open-path'
8+
args:
9+
path: 01_flux_tutorial.ipynb
10+
icon: ./flux-icon.png
11+
catalog: Notebook
12+
13+
- title: Flux Framework Portal
14+
description: Flux Framework portal for projects, releases, and publication.
15+
source: https://flux-framework.org/
16+
type: url
17+
catalog: Flux Resources
18+
args:
19+
sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
20+
- title: Flux Documentation
21+
source: https://flux-framework.readthedocs.io/en/latest/
22+
type: url
23+
catalog: Flux Resources
24+
args:
25+
sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
26+
- title: Flux Cheat Sheet
27+
source: https://flux-framework.org/cheat-sheet/
28+
type: url
29+
catalog: Flux Resources
30+
args:
31+
sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
32+
- title: Flux Glossary of Terms
33+
source: https://flux-framework.readthedocs.io/en/latest/glossary.html
34+
type: url
35+
catalog: Flux Resources
36+
args:
37+
sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
38+
- title: Flux Comics
39+
source: https://flux-framework.readthedocs.io/en/latest/comics/fluxonomicon.html
40+
description: come and meet FluxBird - the pink bird who knows things!
41+
type: url
42+
catalog: Flux Resources
43+
args:
44+
sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
45+
- title: Flux Learning Guide
46+
source: https://flux-framework.readthedocs.io/en/latest/guides/learning_guide.html
47+
description: learn about what Flux does, how it works, and real research applications
48+
type: url
49+
catalog: Flux Resources
50+
args:
51+
sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
52+
- title: Getting Started with Flux and Go
53+
source: https://converged-computing.github.io/flux-go
54+
type: url
55+
catalog: Flux Resources
56+
args:
57+
sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']
58+
- title: Getting Started with Flux in C
59+
source: https://converged-computing.github.io/flux-c-examples/
60+
description: ...looking for contributors!
61+
type: url
62+
catalog: Flux Resources
63+
args:
64+
sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups']

‎tutorial/notebook/docker/start.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
#!/bin/bash
2+
/usr/bin/flux start --test-size=4 /usr/local/bin/jupyter-lab --ip=0.0.0.0

0 commit comments

Comments
 (0)