Skip to content

Commit cf23722

Browse files
maxb-iorspeedbadmin
maxb-ior
and
speedbadmin
authored
716 add build tests for macos and windows to ci pipeline (#764)
* renamed ci to build workflows * ci_pipeline - added new jobs with reusable workflows with build tests on windows, macos and ubuntu arm * ci_pipeline - added new jobs with reusable workflows with build tests on windows, macos and ubuntu arm * switching from reusable workflows to local, added build and tests for Windows, MacOS and Ubuntu on Arm to the ci_workflow * commented out arena_test in the windows workflow * Renamed some jobs * Renamed some jobs * commented out the test for Windows, leaving build only * re-arranges the ci workflow and added a summary job which will be made as required check * updated condition of build and other jobs that come after check license * updated condition of build and other jobs that come after check license * updated condition of build and other jobs that come after check license * fix instructions for macos_Arm * upgraded gh cache action from v2 to v3.3.2 * merged the updated check license and history workflow and disabled condition for it execution from ci_pipeline that made sure it is running only on PR review * switched github.ref_name to github.ref for Linux on Arm build * added a test that checks the workflow trigger and sets up the correct branch for building on Linux Arm, this test is needed for docker container that we use for the build * added Dump GitHub context step to Linux-Arm-build * added case pull_request_review for build on Linux Arm * added case pull_request_review for build on Linux Arm * added case pull_request_review for build on Linux Arm --------- Co-authored-by: speedbadmin <105581954+speedbadmin@users.noreply.github.com>
1 parent 1c6a506 commit cf23722

File tree

6 files changed

+276
-17
lines changed

6 files changed

+276
-17
lines changed

‎.github/workflows/build_and_publish_jar.yml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,12 +30,13 @@ jobs:
3030
run: |
3131
major=$(grep '_MAJOR\s\+[0-9]\+' "$VERSION_FILE" | sed 's/[^0-9]\+//') && echo $major
3232
minor=$(grep '_MINOR\s\+[0-9]\+' "$VERSION_FILE" | sed 's/[^0-9]\+//') && echo $minor
33-
patch=$(( $(grep '_PATCH\s\+[0-9]\+' "$VERSION_FILE" | sed 's/[^0-9]\+//') + 1 )) && echo $patch
33+
#patch=$(( $(grep '_PATCH\s\+[0-9]\+' "$VERSION_FILE" | sed 's/[^0-9]\+//') + 1 )) && echo $patch
34+
patch=$(( $(grep '_PATCH\s\+[0-9]\+' "$VERSION_FILE" | sed 's/[^0-9]\+//'))) && echo $patch
3435
echo "verSion=$major.$minor.$patch" >> $GITHUB_OUTPUT
3536
3637
Mac_i86:
3738
needs: pre_build
38-
uses: ./.github/workflows/ci_macos.yml
39+
uses: ./.github/workflows/build_macos.yml
3940
with:
4041
verSion: ${{ needs.pre_build.outputs.out1 }}
4142
secrets:
@@ -45,19 +46,19 @@ jobs:
4546

4647
Mac_ARM:
4748
needs: pre_build
48-
uses: ./.github/workflows/ci_macos_ARM.yml
49+
uses: ./.github/workflows/build_macos_ARM.yml
4950
with:
5051
verSion: ${{ needs.pre_build.outputs.out1 }}
5152

5253
Ubuntu_ARM:
5354
needs: pre_build
54-
uses: ./.github/workflows/ci_ubuntu_arm.yml
55+
uses: ./.github/workflows/build_ubuntu_arm.yml
5556
with:
5657
verSion: ${{ needs.pre_build.outputs.out1 }}
5758

5859
Windows:
5960
needs: pre_build
60-
uses: ./.github/workflows/ci_windows.yml
61+
uses: ./.github/workflows/build_windows.yml
6162
with:
6263
verSion: ${{ needs.pre_build.outputs.out1 }}
6364
secrets:

.github/workflows/ci_macos.yml renamed to .github/workflows/build_macos.yml

File renamed without changes.

.github/workflows/ci_macos_ARM.yml renamed to .github/workflows/build_macos_ARM.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,12 @@ jobs:
2626
echo "the built library java/target/libspeedbjni-osx-x86_64.jnilib needs to be uploaded to the folloing location"
2727
echo "java/target/libspeedbjni-osx-arm64.jnilib s3://spdb-builder/jar_test/v${{ inputs.verSion }}/libspeedbjni-osx-arm64.jnilib"
2828
29-
# export JAVA_HOME=/usr/local/opt/openjdk@11
30-
# export CPPFLAGS="-I/usr/local/opt/openjdk@11/include"
31-
# export CXXFLAGS="-I/usr/local/opt/openjdk@11/include"
29+
# export JAVA_HOME=`/usr/libexec/java_home -v 11`
30+
# export CPPFLAGS="-I$JAVA_HOME/include"
31+
# export CXXFLAGS="-I$JAVA_HOME/include"
3232
# ROCKSDB_DISABLE_JEMALLOC=1 PORTABLE=1 DEBUG_LEVEL=0 make -j 4 rocksdbjavastatic
3333
3434
- name: 'upload artifacts' #This step executed only when this workflow is called by another and a version is provided
3535
if: inputs.verSion != ' '
3636
run: echo "nothing to do here until the real environment is ready"
37-
#run: aws s3 cp java/target/libspeedbjni-osx-x86_64.jnilib s3://spdb-builder/jar_test/v${{ inputs.verSion }}/libspeedbjni-osx-x86_64.jnilib
37+
#run: aws s3 cp java/target/libspeedbjni-osx-x86_64.jnilib s3://spdb-builder/jar_test/v${{ inputs.verSion }}/libspeedbjni-osx-arm64.jnilib

.github/workflows/ci_ubuntu_arm.yml renamed to .github/workflows/build_ubuntu_arm.yml

File renamed without changes.

.github/workflows/ci_windows.yml renamed to .github/workflows/build_windows.yml

File renamed without changes.

‎.github/workflows/ci_pipeline.yml‎

Lines changed: 266 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,14 @@ permissions: write-all
1313
jobs:
1414
#Sanity:
1515
#uses: speedb-io/speedb/.github/workflows/sanity_check.yml@main
16+
17+
Check-Licence-And-History:
18+
#if: ${{ github.event_name == 'pull_request_review' }}
19+
uses: ./.github/workflows/check_license_and_history.yml
1620

1721
Build:
18-
#needs: [Sanity]
19-
if: ${{ github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release') }}
22+
needs: [Check-Licence-And-History]
23+
if: ${{ (always() && !failure() && !cancelled()) && (github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release')) }}
2024
#runs-on: [self-hosted, ubuntu, asrunner]
2125
runs-on: ubuntu-20.04
2226
strategy:
@@ -58,7 +62,7 @@ jobs:
5862
5963
6064
- name: ccache cache files
61-
uses: actions/cache@v2
65+
uses: actions/cache@v3.3.2
6266
with:
6367
path: ~/.ccache
6468
key: ${{runner.os}}-ccache-${{steps.ccache_cache_timestamp.outputs.timestamp}}
@@ -104,15 +108,269 @@ jobs:
104108
#uses: speedb-io/speedb/.github/workflows/perf-test.yml@main
105109

106110
QA-Tests:
107-
if: ${{ github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release') }}
111+
if: ${{ (always() && !failure() && !cancelled()) && (github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release')) }}
108112
needs: [Build]
109113
uses: speedb-io/speedb/.github/workflows/qa-tests.yml@main
110114

111115
Fuzz:
112-
if: ${{ github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release') }}
116+
if: ${{ (always() && !failure() && !cancelled()) && (github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release')) }}
113117
needs: [Build]
114118
uses: ./.github/workflows/test_fuzz.yml
115119

116-
Check-Licence-And-History:
117-
if: ${{ github.event_name == 'pull_request_review' }}
118-
uses: ./.github/workflows/check_license_and_history.yml
120+
Windows-build-test:
121+
if: ${{ (always() && !failure() && !cancelled()) && (github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release')) }}
122+
needs: [Build]
123+
runs-on: windows-2022
124+
env:
125+
THIRDPARTY_HOME: C:/Users/runneradmin/thirdparty
126+
CMAKE_HOME: C:/Program Files/CMake
127+
CMAKE_BIN: C:/Program Files/CMake/bin/cmake.exe
128+
SNAPPY_HOME: C:/Users/runneradmin/thirdparty/snappy-1.1.9
129+
SNAPPY_INCLUDE: C:/Users/runneradmin/thirdparty/snappy-1.1.9;C:/Users/circleci/thirdparty/snappy-1.1.9/build
130+
SNAPPY_LIB_DEBUG: C:/Users/runneradmin/thirdparty/snappy-1.1.9/build/Debug/snappy.lib
131+
CMAKE_GENERATOR: Visual Studio 17 2022
132+
CODE_HOME: C:/Users/runneradmin/code
133+
134+
steps:
135+
- uses: actions/checkout@v3
136+
137+
- name: Set up JDK 17
138+
uses: actions/setup-java@v3
139+
with:
140+
distribution: 'temurin' # See 'Supported distributions' for available options
141+
java-version: '8'
142+
143+
- name: Add msbuild to PATH
144+
uses: microsoft/setup-msbuild@v1.1
145+
146+
- name: Setup VS Dev
147+
uses: seanmiddleditch/gha-setup-vsdevenv@v4
148+
149+
- name: install cmake
150+
shell: powershell
151+
run: |
152+
echo "Installing CMake..."
153+
choco install cmake --installargs 'ADD_CMAKE_TO_PATH=System' -y
154+
- name: prepare Thirdparty home
155+
shell: powershell
156+
run: mkdir "$Env:THIRDPARTY_HOME"
157+
158+
- name: install snappy test
159+
shell: powershell
160+
run: |
161+
mkdir $env:CODE_HOME
162+
cd $env:CODE_HOME
163+
curl https://github.com/google/snappy/archive/refs/tags/1.1.9.zip -o 1.1.9.zip
164+
Expand-Archive -Path 1.1.9.zip -DestinationPath snappy-tmp
165+
mv .\snappy-tmp\snappy-1.1.9\ .
166+
rmdir .\snappy-tmp\
167+
cd .\snappy-1.1.9\
168+
mkdir build
169+
cd .\build\
170+
cmake -G "Visual Studio 17 2022" -A x64 -DCMAKE_GENERATOR_PLATFORM=x64 -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF ..
171+
msbuild Snappy.sln /p:Configuration=Debug /p:Platform=x64
172+
msbuild Snappy.sln /p:Configuration=Release /p:Platform=x64
173+
174+
- name: install snappy
175+
shell: powershell
176+
run: |
177+
cd $Env:THIRDPARTY_HOME
178+
curl https://github.com/google/snappy/archive/refs/tags/1.1.9.zip -O snappy-1.1.9.zip
179+
Expand-Archive -Path snappy-1.1.9.zip -DestinationPath snappy-tmp
180+
mv .\snappy-tmp\snappy-1.1.9\ .
181+
cd snappy-1.1.9
182+
mkdir build
183+
cd .\build
184+
& cmake -G "Visual Studio 17 2022" -A x64 -DCMAKE_GENERATOR_PLATFORM=x64 -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF ..
185+
msbuild.exe Snappy.sln -maxCpuCount -property:Configuration=Release -property:Platform=x64
186+
187+
- name: install gflags
188+
shell: powershell
189+
run: |
190+
cd $Env:THIRDPARTY_HOME
191+
curl https://github.com/gflags/gflags/archive/refs/tags/v2.2.2.zip -o v2.2.2.zip
192+
Expand-Archive -Path .\v2.2.2.zip -DestinationPath gflags-tmp
193+
mv .\gflags-tmp\gflags-2.2.2 .
194+
rmdir gflags-tmp
195+
cd gflags-2.2.2
196+
mkdir target
197+
cd target
198+
cmake -G "Visual Studio 17 2022" -A x64 ..
199+
msbuild gflags.sln /p:Configuration=Release /p:Platform=x64
200+
- name: install zlib
201+
shell: powershell
202+
run: |
203+
cd $Env:THIRDPARTY_HOME
204+
curl https://zlib.net/zlib13.zip -o zlib13.zip
205+
Expand-Archive -Path zlib13.zip -DestinationPath zlib-tmp
206+
mv .\zlib-tmp\zlib-1.3\ .
207+
rmdir zlib-tmp
208+
cd zlib-1.3\contrib\vstudio\vc14
209+
devenv zlibvc.sln /upgrade
210+
cp ../../../zlib.h .
211+
msbuild zlibvc.sln /p:Configuration=Debug /p:Platform=x64
212+
msbuild zlibvc.sln /p:Configuration=Release /p:Platform=x64
213+
copy x64\ZlibDllRelease\zlibwapi.lib x64\ZlibStatRelease\
214+
- name: install lz4
215+
shell: powershell
216+
run: |
217+
cd $Env:THIRDPARTY_HOME
218+
curl https://github.com/lz4/lz4/archive/refs/tags/v1.9.2.zip -o lz4.zip
219+
Expand-Archive -Path lz4.zip -DestinationPath lz4-tmp
220+
mv .\lz4-tmp\lz4-1.9.2\ .
221+
rmdir .\lz4-tmp\
222+
cd .\lz4-1.9.2\
223+
cd visual\VS2017
224+
devenv lz4.sln /upgrade
225+
msbuild lz4.sln /p:Configuration=Release /p:Platform=x64
226+
- name: install zctd
227+
shell: powershell
228+
run: |
229+
cd $Env:THIRDPARTY_HOME
230+
curl https://github.com/facebook/zstd/archive/v1.5.2.zip -o zstd-tmp.zip
231+
Expand-Archive -Path zstd-tmp.zip -DestinationPath zstd-tmp
232+
mv .\zstd-tmp\zstd-1.5.2\ .
233+
rmdir .\zstd-tmp\
234+
cd zstd-1.5.2\build\VS2010
235+
devenv zstd.sln /upgrade
236+
msbuild zstd.sln /p:Configuration=Debug /p:Platform=x64
237+
msbuild zstd.sln /p:Configuration=Release /p:Platform=x64
238+
239+
- name: Build Speedb
240+
run: |
241+
copy C:\Users\runneradmin\thirdparty\snappy-1.1.9\build\snappy-stubs-public.h C:\Users\runneradmin\thirdparty\snappy-1.1.9\
242+
copy tools\thirdparty.txt thirdparty.inc # copy the thirdparty.inc that reflects the env on the runner machine
243+
mkdir build
244+
cd build
245+
& $Env:CMAKE_BIN -G "$Env:CMAKE_GENERATOR" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_STANDARD=20 -A x64 -DJNI=1 -DGFLAGS=1 -DOPTDBG=1 -DPORTABLE=1 -DSNAPPY=1 -DJNI=1 -DSNAPPY=1 -DLZ4=1 -DZLIB=1 -DZSTD=1 -DXPRESS=1 -DFAIL_ON_WARNINGS=0 ..
246+
cd ..
247+
echo "Building with VS version: $Env:CMAKE_GENERATOR"
248+
msbuild build/speedb.sln /p:Configuration=Release /t:speedbjni-shared
249+
#msbuild.exe build/speedb.sln -maxCpuCount -property:Configuration=Debug -property:Platform=x64
250+
251+
- name: Test Speedb
252+
shell: powershell
253+
run: |
254+
echo "skipping all tests for now"
255+
#build_tools\run_ci_db_test.ps1 -SuiteRun arena_test,db_basic_test,db_test,db_test2,db_merge_operand_test,bloom_test,c_test,coding_test,crc32c_test,dynamic_bloom_test,env_basic_test,env_test,hash_test,random_test -Concurrency 16
256+
#build_tools\run_ci_db_test.ps1 -SuiteRun db_test,db_test2,db_merge_operand_test,bloom_test,c_test,coding_test,crc32c_test,dynamic_bloom_test,env_basic_test,env_test,hash_test,random_test -Concurrency 16
257+
258+
259+
260+
Macos-build:
261+
if: ${{ (always() && !failure() && !cancelled()) && (github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release')) }}
262+
needs: [Build]
263+
runs-on: macos-11
264+
steps:
265+
- uses: actions/checkout@v3
266+
- name: Set up JDK 17
267+
uses: actions/setup-java@v3
268+
with:
269+
distribution: 'temurin' # See 'Supported distributions' for available options
270+
java-version: '8'
271+
- name: build jar
272+
run: |
273+
echo $JAVA_HOME
274+
export CPPFLAGS="-I$JAVA_HOME/include"
275+
export CXXFLAGS="-I$JAVA_HOME/include"
276+
brew install zlib
277+
brew install bzip2 lz4 snappy
278+
ROCKSDB_DISABLE_JEMALLOC=1 PORTABLE=1 DEBUG_LEVEL=0 make -j 4 rocksdbjavastatic
279+
280+
281+
Linux-build:
282+
if: ${{ (always() && !failure() && !cancelled()) && (github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release')) }}
283+
needs: [Build]
284+
runs-on: ubuntu-latest
285+
286+
container:
287+
image: centos:7.9.2009
288+
289+
steps:
290+
- name: pre
291+
run: |
292+
yum install -y centos-release-scl epel-release
293+
yum install -y make devtoolset-11-gcc-c++ \
294+
coreutils wget unzip which git python3 openssl openssl-devel \
295+
libzstd-devel lz4-devel snappy-devel zlib-devel readline-devel \
296+
java-1.8.0-openjdk-devel
297+
echo "PATH=/opt/rh/devtoolset-11/root/usr/bin:${PATH}" >> $GITHUB_ENV
298+
echo "RELEASE_VERSION=${GITHUB_REF_NAME#speedb/v}" >> $GITHUB_ENV
299+
300+
- name: Install CMake
301+
run: |
302+
CMAKE_RELEASE=3.20.1
303+
wget https://github.com/Kitware/CMake/releases/download/v${CMAKE_RELEASE}/cmake-${CMAKE_RELEASE}.tar.gz
304+
tar xf cmake-${CMAKE_RELEASE}.tar.gz
305+
cd cmake-${CMAKE_RELEASE}
306+
./bootstrap
307+
make -j$(nproc) && make install
308+
309+
- uses: actions/checkout@v3
310+
311+
- run: mkdir "$GITHUB_WORKSPACE/out"
312+
313+
- name: Build and package release libraries
314+
run: |
315+
rm -rf build && mkdir build && cd build
316+
cmake .. -DCMAKE_BUILD_TYPE=Release -DSPDB_RELEASE_BUILD=1 -DPORTABLE=1 -DWITH_GFLAGS=0 -DWITH_SNAPPY=1 -DWITH_LZ4=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1
317+
318+
- name: Build Jar
319+
run: |
320+
make clean
321+
SPDB_RELEASE_BUILD=1 LIB_MODE=static DEBUG_LEVEL=0 PORTABLE=1 JAVA_HOME=/usr/lib/jvm/java-openjdk make -j$(nproc) rocksdbjavastatic
322+
323+
- name: Build db_bench
324+
run: |
325+
yum install -y gflags-devel
326+
rm -rf build && mkdir build && cd build
327+
cmake .. -DCMAKE_BUILD_TYPE=Release -DSPDB_RELEASE_BUILD=1 -DPORTABLE=1 -DWITH_GFLAGS=1 \
328+
-DWITH_SNAPPY=1 -DWITH_LZ4=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 \
329+
-DWITH_BENCHMARK_TOOLS=1 -DROCKSDB_BUILD_SHARED=1
330+
make -j$(nproc) db_bench
331+
332+
333+
Linux-Arm-build:
334+
if: ${{ (always() && !failure() && !cancelled()) && (github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release')) }}
335+
needs: [Build]
336+
runs-on: ubuArm64G
337+
steps:
338+
- name: Dump GitHub context
339+
env:
340+
GITHUB_CONTEXT: ${{ toJson(github) }}
341+
run: |
342+
echo "$GITHUB_CONTEXT"
343+
344+
- name: 'Checkout GitHub Action'
345+
uses: actions/checkout@v3
346+
347+
- name: 'Build_on_Arm_Centos'
348+
run: |
349+
export SPDB_LIB_DIR=~/spdb_lib && mkdir -p $SPDB_LIB_DIR
350+
case "$GITHUB_EVENT_NAME" in
351+
"pull_request")
352+
echo "this workflow was triggered by a pull request"
353+
echo "the branch can not be used, it is $GITHUB_REF_NAME , instead ${{ github.head_ref }} will be used"
354+
docker run --rm -v $(readlink -f ${SPDB_LIB_DIR}):/out -i speedb-centos-builder ${{ github.head_ref }}
355+
;;
356+
"workflow_dispatch")
357+
echo "this workflow was triggered by a workflow dispatch, we will use the ref_name instead of the merge branch"
358+
echo " consider using github.ref, the branch that will be used here: " ${{ github.ref_name }}
359+
docker run --rm -v $(readlink -f ${SPDB_LIB_DIR}):/out -i speedb-centos-builder ${{ github.ref_name }}
360+
;;
361+
"pull_request_review")
362+
echo "this workflow was triggered by a pull request review"
363+
echo "the branch can not be used, it is $GITHUB_REF_NAME , instead ${{ github.event.pull_request.head.ref }} will be used"
364+
docker run --rm -v $(readlink -f ${SPDB_LIB_DIR}):/out -i speedb-centos-builder ${{ github.event.pull_request.head.ref }}
365+
;;
366+
esac
367+
368+
CI-all:
369+
if: ${{ github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release') }}
370+
needs: [Check-Licence-And-History, Build, QA-Tests, Fuzz, Linux-Arm-build, Linux-build, Macos-build, Windows-build-test]
371+
runs-on: ubuntu-latest
372+
steps:
373+
- name: Summary
374+
run: |
375+
echo "All tests passed"
376+
exit 0

0 commit comments

Comments
 (0)