Update to v0.4.2

Update python package and documentation from v0.4.1 to v0.4.2.

[ committed by @ashao ]
[ reviewed by @MattToast ]

Author: ashao

.github/workflows/build_docs.yml

@@ -1,3 +1,31 @@
+#
+# BSD 2-Clause License
+#
+# Copyright (c) 2021-2023, Hewlett Packard Enterprise
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this
+#    list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+#    this list of conditions and the following disclaimer in the documentation
+#    and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+
 name: deploy_dev_docs
 
 on:

.github/workflows/release.yml

@@ -1,3 +1,31 @@
+#
+# BSD 2-Clause License
+#
+# Copyright (c) 2021-2023, Hewlett Packard Enterprise
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this
+#    list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+#    this list of conditions and the following disclaimer in the documentation
+#    and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+
 name: deploy-release
 
 on:

.github/workflows/run_tests.yml

@@ -1,3 +1,31 @@
+#
+# BSD 2-Clause License
+#
+# Copyright (c) 2021-2023, Hewlett Packard Enterprise
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this
+#    list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+#    this list of conditions and the following disclaimer in the documentation
+#    and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+
 name: run-tests
 
 on:
@@ -23,10 +51,15 @@ jobs:
       matrix:
         os: [macos-10.15, ubuntu-20.04] # Operating systems
         compiler: [8] # GNU compiler version
-        rai: [1.2.3, 1.2.5] # Redis AI versions
-        py_v: [3.7, 3.8, 3.9] # Python versions
+        rai: [1.2.5, 1.2.7] # Redis AI versions
+        py_v: [3.8, 3.9, '3.10'] # Python versions
         exclude:
-          - os: macos-10.15 # Do not build with Redis AI 1.2.5 on MacOS
+          # Do not build with Redis AI 1.2.5 on MacOS
+          - os: macos-10.15
+            rai: 1.2.5
+          # Do not build Redis AI 1.2.5 with py3.10
+          # as wheels for dependecies are not availble
+          - py_v: '3.10'
             rai: 1.2.5
 
     env:
@@ -70,20 +103,20 @@ jobs:
         if: contains( matrix.os, 'ubuntu' ) && matrix.py_v == 3.9 && matrix.rai == '1.2.5'
         run: singularity pull docker://alrigazzi/smartsim-testing
 
+      # Note: The develop branch of smartredis is installed first to ensure that any tests that depend
+      # on developments of the client are brought in.
       - name: Install SmartSim (with ML backends)
-        run: python -m pip install .[dev,ml,ray]
-
-      - name: Install ML Runtimes with Smart
-        if: contains( matrix.os, 'macos' )
-        run: smart build --device cpu -v
+        run: |
+          python -m pip install git+https://github.com/CrayLabs/SmartRedis.git@develop#egg=smartredis
+          python -m pip install .[dev,ml]
 
       - name: Install ML Runtimes with Smart (with pt, tf, and onnx support)
-        if: contains( matrix.os, 'ubuntu' ) && (matrix.py_v != 3.9 || matrix.rai != '1.2.3')
+        if: (matrix.py_v != '3.10')
         run: smart build --device cpu --onnx -v
 
-      - name: Install ML Runtimes with Smart excluding PyTorch for Ubuntu/Python3.9/RAI1.2.3 combo
-        if: contains( matrix.os, 'ubuntu' ) && matrix.py_v == 3.9 && matrix.rai == '1.2.3'
-        run: smart build --device cpu --no_pt --onnx -v
+      - name: Install ML Runtimes with Smart (with pt and tf support)
+        if: (matrix.py_v == '3.10')
+        run: smart build --device cpu -v
 
       - name: Run Pytest
         run: |

.gitignore

@@ -1,4 +1,5 @@
 .vscode
+**/*.swp
 __pycache__
 .ipynb_checkpoints
 .pytest_cache/
@@ -22,4 +23,4 @@ smartsim/_core/bin/*-server
 smartsim/_core/bin/*-cli
 
 # created upon install
-smartsim/_core/lib
+smartsim/_core/lib

.wci.yml

@@ -0,0 +1,48 @@
+---
+    # Metadata parsed by the workflow community initiative, https://workflows.community/systems
+    # Registered at https://github.com/workflowscommunity/workflowscommunity.github.io/blob/main/_data/workflow_systems.yml
+
+    name: SmartSim
+
+    headline: Machine learning workflows with HPC applications (Python, C++, C, and Fortran)
+
+    description: SmartSim is a workflow library that makes it easier to use common
+      Machine Learning (ML) libraries, like PyTorch and TensorFlow,
+      in combination with High Performance Computing (HPC) simulations and applications.
+      SmartSim launches ML infrastructure on HPC systems alongside user workloads
+      and supports most HPC workload managers (e.g. Slurm, PBSPro, LSF, Cobalt).
+      SmartSim also provides a set of client libraries in Python, C++, C, and Fortran.
+      These client libraries allow users to send and receive data between user
+      applications and the machine learning infrastructure.  Moreover, the
+      client APIs enable the execution of machine learning tasks like inference
+      and online training from within user code.  The exchange of data and
+      execution of machine learning tasks is orchestrated by a high performance
+      in-memory database that is launched and managed by SmartSim.
+
+    language: Python
+
+    release:
+      version: 0.4.2
+      date: 2023-04-12
+
+    documentation:
+      general: https://www.craylabs.org/docs/overview.html
+      installation: https://www.craylabs.org/docs/installation.html
+      tutorial: https://www.craylabs.org/docs/tutorials/getting_started/getting_started.html
+
+    execution_environment:
+      interfaces:
+        - Python API
+        - Python Client API
+        - C++ Client API
+        - C Client API
+        - Fortran Client API
+      resource_managers:
+        - Slurm
+        - PBSPro
+        - LSF
+        - Cobalt
+        - Linux/MacOS
+      transfer_protocols:
+        - TCP/IP
+        - Unix Domain Sockets (UDS)

LICENSE.md

@@ -1,6 +1,6 @@
 BSD 2-Clause License
 
-Copyright (c) 2021-2022, Hewlett Packard Enterprise
+Copyright (c) 2021-2023, Hewlett Packard Enterprise
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

Makefile

@@ -1,3 +1,29 @@
+# BSD 2-Clause License
+#
+# Copyright (c) 2021-2023, Hewlett Packard Enterprise
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# 1. Redistributions of source code must retain the above copyright notice, this
+#    list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+#    this list of conditions and the following disclaimer in the documentation
+#    and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 
 MAKEFLAGS += --no-print-directory
 
@@ -114,11 +140,11 @@ tutorials-dev:
 	@docker compose build tutorials-dev
 	@docker run -p 8888:8888 smartsim-tutorials:dev-latest
 
-# help: tutorials-prod                 - Build and start a docker container to run the tutorials (v0.4.1)
+# help: tutorials-prod                 - Build and start a docker container to run the tutorials (v0.4.2)
 .PHONY: tutorials-prod
 tutorials-prod:
 	@docker compose build tutorials-prod
-	@docker run -p 8888:8888 smartsim-tutorials:v0.4.1
+	@docker run -p 8888:8888 smartsim-tutorials:v0.4.2
 
 
 # help:

README.md

@@ -16,6 +16,8 @@
 </div>
 
 
+<div align="center">
+
 [![License](https://img.shields.io/github/license/CrayLabs/SmartSim)](https://github.com/CrayLabs/SmartSim/blob/master/LICENSE.md)
 ![GitHub last commit](https://img.shields.io/github/last-commit/CrayLabs/SmartSim)
 ![GitHub deployments](https://img.shields.io/github/deployments/CrayLabs/SmartSim/github-pages?label=doc%20build)
@@ -27,6 +29,8 @@
 [![codecov](https://codecov.io/gh/CrayLabs/SmartSim/branch/develop/graph/badge.svg?token=96HFI2F45E)](https://codecov.io/gh/CrayLabs/SmartSim)
 [![Downloads](https://static.pepy.tech/personalized-badge/smartsim?period=total&units=international_system&left_color=grey&right_color=orange&left_text=Downloads)](https://pepy.tech/project/smartsim)
 
+</div>
+
 ------------
 
 # SmartSim
@@ -69,8 +73,6 @@ exchanged between applications at runtime without the utilization of MPI.
     - [Local Launch](#local-launch)
     - [Interactive Launch](#interactive-launch)
     - [Batch Launch](#batch-launch)
-  - [Ray](#ray)
-    - [Ray on HPC](#ray-on-hpc)
 - [SmartRedis](#smartredis)
   - [Tensors](#tensors)
   - [Datasets](#datasets)
@@ -97,8 +99,8 @@ before using it on your system. Each tutorial is a Jupyter notebook that can be
 which will run a jupyter lab with the tutorials, SmartSim, and SmartRedis installed.
 
 ```bash
-docker pull ghcr.io/craylabs/smartsim-tutorials:v0.4.1
-docker run -p 8888:8888 ghcr.io/craylabs/smartsim-tutorials:v0.4.1
+docker pull ghcr.io/craylabs/smartsim-tutorials:v0.4.2
+docker run -p 8888:8888 ghcr.io/craylabs/smartsim-tutorials:v0.4.2
 # click on link to open jupyter lab
 ```
 
@@ -284,7 +286,6 @@ initialization. Local launching does not support batch workloads.
 
 # Infrastructure Library Applications
  - Orchestrator - In-memory data store and Machine Learning Inference (Redis + RedisAI)
- - Ray - Distributed Reinforcement Learning (RL), Hyperparameter Optimization (HPO)
 
 ## Redis + RedisAI
 
@@ -398,53 +399,6 @@ exp.stop(db_cluster)
 python run_db_batch.py
 ```
 
------
-## Ray
-
-Ray is a distributed computation framework that supports a number of applications
- - RLlib - Distributed Reinforcement Learning (RL)
- - RaySGD - Distributed Training
- - Ray Tune - Hyperparameter Optimization (HPO)
- - Ray Serve - ML/DL inference
-As well as other integrations with frameworks like Modin, Mars, Dask, and Spark.
-
-Historically, Ray has not been well supported on HPC systems. A few examples exist,
-but none are well maintained. Because SmartSim already has launchers for HPC systems,
-launching Ray through SmartSim is a relatively simple task.
-
-### Ray on HPC
-
-Below is an example of how to launch a Ray cluster on an HPC system and connect to it.
-In this example, we set `batch=True`, which means that the cluster will be started
-requesting an allocation through the scheduler (Slurm, PBS, etc). If this code
-is run within a sufficiently large interactive allocation, setting `batch=False`
-will spin the Ray cluster on the allocated nodes.
-
-```Python
-import ray
-
-from smartsim import Experiment
-from smartsim.exp.ray import RayCluster
-
-exp = Experiment("ray-cluster", launcher='auto')
-# 3 workers + 1 head node = 4 node-cluster
-cluster = RayCluster(name="ray-cluster", run_args={},
-                     ray_args={"num-cpus": 24},
-                     launcher='auto', num_nodes=4, batch=True)
-
-exp.generate(cluster, overwrite=True)
-exp.start(cluster, block=False, summary=True)
-
-# Connect to the Ray cluster
-ctx = ray.init(f"ray://{cluster.get_head_address()}:10001")
-
-# <run Ray tune, RLlib, HPO...>
-```
-
-*New in 0.4.0* the auto argument enables the Ray Cluster to be launched
-across scheduler types. Both batch launch and interactive launch commands
-will be automatically detected and used by SmartSim.
-
 ------
 # SmartRedis
 
@@ -498,7 +452,7 @@ which will run a jupyter lab with the tutorials, SmartSim, and SmartRedis instal
 
 ```bash
 docker pull ghcr.io/craylabs/smartsim-tutorials:v1
-docker run -p 8888:8888 ghcr.io/craylabs/smartsim-tutorials:v0.4.1
+docker run -p 8888:8888 ghcr.io/craylabs/smartsim-tutorials:v0.4.2
 ```
 Each of the following examples can be found in the
 [SmartSim documentation](https://www.craylabs.org/docs/tutorials/getting_started/getting_started.html).
@@ -683,17 +637,17 @@ from C, C++, Fortran and Python with the SmartRedis Clients:
   </thead>
   <tbody style="text-align:center">
     <tr>
-      <td rowspan="3">1.2.3-1.2.4</td>
+      <td rowspan="3">1.2.7</td>
       <td>PyTorch</td>
-      <td>1.7.x</td>
+      <td>1.11.x</td>
     </tr>
     <tr>
       <td>TensorFlow\Keras</td>
-      <td>2.4.x-2.5.x</td>
+      <td>2.8.x</td>
     </tr>
     <tr>
       <td>ONNX</td>
-      <td>1.9.x</td>
+      <td>1.11.x</td>
     </tr>
       <td rowspan="3">1.2.5</td>
       <td>PyTorch</td>