Skip to content

Commit 6c0109b

Browse files
author
joeylee.lz
committed
Add FrameworkBenchmarks performance test
1 parent c671a7f commit 6c0109b

File tree

5 files changed

+226
-0
lines changed

5 files changed

+226
-0
lines changed

‎perf/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ subdirectory and given a meaningful name. Once the reorganization of this direc
2626
//├── dacapo
2727
//├── renaissance
2828
//├── liberty
29+
//├── FrameworkBenchmarks
2930
3031
```
3132
Each subdirectory requires a build.xml file describing where to pull the benchmark suite from, and how to build and run it. Each subdirectory also requires a playlist.xml file which describes 1 or more benchmarks and what commands to run in order to execute a particular benchmark run.
@@ -49,4 +50,7 @@ Liberty benchmarks from https://github.com/OpenLiberty - including liberty-dt7-s
4950
#### renaissance
5051
Renaissance benchmarks from https://github.com/renaissance-benchmarks/renaissance - including renaissance-akka-uct, renaissance-als, renaissance-chi-square, renaissance-db-shootout, renaissance-dec-tree, renaissance-finagle-chirper, renaissance-finagle-http, renaissance-fj-kmeans, renaissance-future-genetic, renaissance-gauss-mix, renaissance-log-regression, renaissance-mnemonics, renaissance-movie-lens, renaissance-naive-bayes, renaissance-par-mnemonics, renaissance-philosophers and renaissance-scala-kmeans
5152

53+
#### FrameworkBenchmarks
54+
FrameworkBenchmarks from https://github.com/TechEmpower/FrameworkBenchmarks - including a wide field of web application frameworks for JAVA.
55+
5256
Additional benchmarks are being reviewed for addition and if you wish to include more, please comment in the open performance benchmarks [issue 1112](https://github.com/adoptium/aqa-tests/issues/1112).

‎perf/framework-benchmarks/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# FrameworkBenchmarks
2+
3+
[FrameworkBenchmarks](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Java) provides representative performance measures across a wide field of web application frameworks. With much help from the community, coverage is quite broad and we are happy to broaden it further with contributions.
4+
5+
Here we focus on the wide range of Java frameworks for web applications scenarios.
6+
7+
FrameworkBenchmarks covers common workloads as plaintext processing, json content parsing, popular database manipulations
8+
for different frameworks.
9+
10+
Notice the different from external tests:
11+
```aidl
12+
external tests runs framework inner unit tests to ensure frameworks behaving correctly.
13+
perf framework tests stresses common worloads to ensure frameworks are performing well under test VMs.
14+
```
15+
16+
We are enabling the following list of frameworks to avoid regression:
17+
18+
| Framework | Workloads |
19+
|-----------|----------------------------------------|
20+
| netty | plaintext,json |
21+
| quarkus | plaintext,json,query,fortune,update,db |
22+
| spark | plaintext,json,query,fortune,update,db |
23+
| spring | plaintext,json,query,fortune,update,db |
24+
| spring-jpa | plaintext,json,query,fortune,update,db |
25+
| servlet | plaintext,json |
26+
| servlet3 | plaintext,json |

‎perf/framework-benchmarks/build.xml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
<?xml version="1.0"?>
2+
<project name="FrameworkBenchmarks-Test" default="build" basedir=".">
3+
<taskdef resource="net/sf/antcontrib/antlib.xml" />
4+
<description>
5+
Build FrameworkBenchmarks-Test Docker image
6+
</description>
7+
8+
<!-- set properties for this build -->
9+
<property name="TEST" value="framework-benchmarks" />
10+
<property name="DEST" value="${BUILD_ROOT}/perf/${TEST}" />
11+
<property name="src" location="." />
12+
13+
<condition property="git-prefix" value="git" else="https">
14+
<isset property="isZOS"/>
15+
</condition>
16+
17+
<target name="init">
18+
<mkdir dir="${DEST}"/>
19+
</target>
20+
21+
<target name="getFrameworkBenchmarks" depends="init" description="Clone the distribution">
22+
<echo message="Cloning FrameworkBenchmarks"/>
23+
<var name="git_command" value="clone --depth 1 -b master ${git-prefix}://github.com/TechEmpower/FrameworkBenchmarks.git ${DEST}"/>
24+
<echo message="git ${git_command}" />
25+
<exec executable="git" failonerror="false" dir=".">
26+
<arg line="${git_command}" />
27+
</exec>
28+
</target>
29+
30+
<target name="dist" depends="getFrameworkBenchmarks" description="generate the distribution">
31+
<copy todir="${DEST}">
32+
<fileset dir="${src}" includes="*.xml, *.mk"/>
33+
</copy>
34+
</target>
35+
36+
<target name="build">
37+
<antcall target="dist" inheritall="true" />
38+
</target>
39+
</project>

‎perf/framework-benchmarks/playlist.xml‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
<?xml version='1.0' encoding='UTF-8'?>
2+
<!--
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
-->
15+
<playlist xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../TKG/playlist.xsd">
16+
<test>
17+
<testCaseName>FrameworkBenchmarks-netty</testCaseName>
18+
<command>
19+
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh netty netty.dockerfile ${TEST_RESROOT}; \
20+
$(TEST_STATUS)
21+
</command>
22+
<levels>
23+
<level>special</level>
24+
</levels>
25+
<groups>
26+
<group>perf</group>
27+
</groups>
28+
<versions>
29+
<version>11+</version>
30+
</versions>
31+
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
32+
</test>
33+
<test>
34+
<testCaseName>FrameworkBenchmarks-quarkus</testCaseName>
35+
<command>
36+
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh quarkus quarkus.dockerfile ${TEST_RESROOT}; \
37+
$(TEST_STATUS)
38+
</command>
39+
<levels>
40+
<level>special</level>
41+
</levels>
42+
<groups>
43+
<group>perf</group>
44+
</groups>
45+
<versions>
46+
<version>11+</version>
47+
</versions>
48+
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
49+
</test>
50+
<test>
51+
<testCaseName>FrameworkBenchmarks-spring</testCaseName>
52+
<command>
53+
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh spring spring.dockerfile ${TEST_RESROOT}; \
54+
$(TEST_STATUS)
55+
</command>
56+
<levels>
57+
<level>special</level>
58+
</levels>
59+
<groups>
60+
<group>perf</group>
61+
</groups>
62+
<versions>
63+
<version>11+</version>
64+
</versions>
65+
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
66+
</test>
67+
<test>
68+
<testCaseName>FrameworkBenchmarks-spring-jpa</testCaseName>
69+
<command>
70+
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh spring spring-jpa.dockerfile ${TEST_RESROOT}; \
71+
$(TEST_STATUS)
72+
</command>
73+
<levels>
74+
<level>special</level>
75+
</levels>
76+
<groups>
77+
<group>perf</group>
78+
</groups>
79+
<versions>
80+
<version>11+</version>
81+
</versions>
82+
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
83+
</test>
84+
<test>
85+
<testCaseName>FrameworkBenchmarks-spark</testCaseName>
86+
<command>
87+
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh spark spark.dockerfile ${TEST_RESROOT}; \
88+
$(TEST_STATUS)
89+
</command>
90+
<levels>
91+
<level>special</level>
92+
</levels>
93+
<groups>
94+
<group>perf</group>
95+
</groups>
96+
<versions>
97+
<version>11+</version>
98+
</versions>
99+
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
100+
</test>
101+
<test>
102+
<testCaseName>FrameworkBenchmarks-servlet3</testCaseName>
103+
<command>
104+
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh servlet3 servlet3.dockerfile ${TEST_RESROOT}; \
105+
$(TEST_STATUS)
106+
</command>
107+
<levels>
108+
<level>special</level>
109+
</levels>
110+
<groups>
111+
<group>perf</group>
112+
</groups>
113+
<versions>
114+
<version>11+</version>
115+
</versions>
116+
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
117+
</test>
118+
<test>
119+
<testCaseName>FrameworkBenchmarks-servlet</testCaseName>
120+
<command>
121+
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh servlet servlet.dockerfile ${TEST_RESROOT}; \
122+
$(TEST_STATUS)
123+
</command>
124+
<levels>
125+
<level>special</level>
126+
</levels>
127+
<groups>
128+
<group>perf</group>
129+
</groups>
130+
<versions>
131+
<version>11+</version>
132+
</versions>
133+
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
134+
</test>
135+
</playlist>

‎perf/tfb-runner.sh‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
test=$1
2+
dockerfile=$2
3+
TEST_RESROOT=$3
4+
5+
cd ${TEST_RESROOT}
6+
testcase=` echo ${dockerfile} | cut -d'.' -f 1`
7+
dockerfile=${TEST_RESROOT}/frameworks/Java/${test}/${dockerfile}
8+
9+
if [ ! -f ${dockerfile} ]; then
10+
echo "Warning: ${dockerfile} not found"
11+
exit 1
12+
fi
13+
14+
cp -r ${TEST_JDK_HOME} ${TEST_RESROOT}/frameworks/Java/${test}/jdk
15+
awk '/CMD.*/ && !x {print "COPY jdk /opt/java/openjdk"; x=1} 1' ${dockerfile} > tmpt && mv tmpt ${dockerfile}
16+
awk '/CMD.*/ && !x {print "ENV JAVA_HOME=/opt/java/openjdk"; x=1} 1' ${dockerfile} > tmpt && mv tmpt ${dockerfile}
17+
awk '/CMD.*/ && !x {print "ENV PATH=/opt/java/openjdk/bin:$PATH"; x=1} 1' ${dockerfile} > tmpt && mv tmpt ${dockerfile}
18+
cat ${dockerfile}
19+
./tfb --test $testcase
20+
21+
# get removable results
22+
docker run --entrypoint /bin/bash --rm --network tfb -v /var/run/docker.sock:/var/run/docker.sock -v ${TEST_RESROOT}:/FrameworkBenchmarks techempower/tfb -c "chmod -R 777 /FrameworkBenchmarks/results"

0 commit comments

Comments
 (0)