AWS and Jenkins

Not finished yet, still deciding what to do

This is assuming you have Jenkins set up already and you have access to the web interface. These instructions are copied directly from the Jenkins configuration I'm using to launch Cfncluster

-On the left side of the home page, select "New Item"

-Enter an item name and select "Freestyle project"

-Enter a description for the project

-Check mark "Discard old builds

-Under Strategy, select "Log Rotation"

-Days to keep builds "30"

-Max # of builds to keep "100"

-Check mark "Github project"

-Project url "https://github.com/open-mpi/aws-pmix-scale-testing/"  //Jenkins pulls code from here

Under "Build Environment"

-Check mark "Delete workspace before build starts"

-Check mark "Abort the build if it's stuck

-Time-out strategy "No Activity"

-timeout seconds "300"

-Time-out actions "Writing the build description"

    -Description "Aborted build: 5 minutes without output"

Under "Build" -In Execute shell, you'll want to put the following commands:

cd /path/to/here/aws-pmix-scale-testing/
/path/to/here/aws-pmix-scale-testing/launch.sh ${BUILD_NUMBER}  //BUILD_NUMBER is a Jenkins variable
sleep 20m
/path/to/here/aws-pmix-scale-testing/exists.sh ${BUILD_NUMBER}
aws s3 cp s3://<logsbucketname>/${BUILD_NUMBER}-0_output.tar.gz /var/lib/jenkins/workspace/AWS.launch.test/

You'll also need to add a new script to the aws-pmix-scale-testing folder. Add the exists.sh script:

#!/bin/bash

source /var/lib/jenkins/aws-pmix-scale-testing/buckets.sh

flag=false
path_to_file="s3://$logs/${1}-0_output.tar.gz"

while [ "$flag" = false ]
do
    exists=$(aws s3 ls $path_to_file)
    if [ -z "$exists" ]; then
        echo "${path_to_file} doesn't exist yet"
        sleep 1m
    else
        echo "${path_to_file} exits"
        echo "Copying to /var/lib/jenkins/workspace/AWS.launch.test/"
        flag=true
    #   Will upload files from Jenkins now 
    #   aws s3 cp s3://$logs/${1}-0_output.tar.gz /var/lib/jenkins/workspace/AWS.launch.test/
    fi
done