Skip to content

Commit 0777811

Browse files
portanteportante
committed
Reduce redundant test setup operations
This is a rework of the unit test infrastructure for bench-scripts to reduce the amount of redundant operations performed for each test. Now, all tests automatically have a `--config=test-##` parameter added to their invocation, and the full pbench-agent.cfg file is used for them as well. We enhance the infrastructure to look for common files that tests might leave as artifacts that should show up on the gold file output. We add additional error checking for creating directories and files during test setup. We correct a bug with pbench-dbench so that it now properly emits the iteration name. We also remove trailing spaces from generated output, and add a missing newline to the user-benchmark JSON file generated to make it look like expected output in the gold files.
1 parent fbb91c1 commit 0777811

File tree

14 files changed

+531
-405
lines changed

14 files changed

+531
-405
lines changed

‎agent/bench-scripts/gold/pbench-dbench/test-02.txt‎

Lines changed: 127 additions & 123 deletions
Large diffs are not rendered by default.

‎agent/bench-scripts/gold/pbench-fio/test-19.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ ERROR: foo must be executable
33
--- Finished test-19 pbench-fio (status=1)
44
+++ pbench tree state
55
/var/tmp/pbench-test-bench/pbench-agent
6+
/var/tmp/pbench-test-bench/pbench-agent/fio_test-19_1900.01.01T00.00.00
7+
/var/tmp/pbench-test-bench/pbench-agent/fio_test-19_1900.01.01T00.00.00/metadata.log
68
/var/tmp/pbench-test-bench/pbench-agent/tmp
79
/var/tmp/pbench-test-bench/pbench-agent/tools-default
810
/var/tmp/pbench-test-bench/pbench-agent/tools-default/mpstat
@@ -13,3 +15,6 @@ grep: /var/tmp/pbench-test-bench/pbench-agent/pbench.log: No such file or direct
1315
--- pbench.log file contents
1416
+++ test-execution.log file contents
1517
--- test-execution.log file contents
18+
+++ fio_test-19_1900.01.01T00.00.00/metadata.log file contents
19+
[pbench]
20+
--- fio_test-19_1900.01.01T00.00.00/metadata.log file contents

‎agent/bench-scripts/gold/pbench-moongen/test-03.txt‎

Lines changed: 138 additions & 133 deletions
Large diffs are not rendered by default.

‎agent/bench-scripts/gold/pbench-user-benchmark/test-09.txt‎

Lines changed: 35 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,21 @@
11
+++ Running test-09 pbench-user-benchmark
22
Running user-benchmark-script no-file
33
--- Finished test-09 pbench-user-benchmark (status=0)
4-
{
5-
"duration":0,
6-
"duration_units":"sec"
7-
}+++ pbench tree state
4+
+++ pbench tree state
85
/var/tmp/pbench-test-bench/pbench-agent
9-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00
10-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1
11-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/reference-result
12-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1
13-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1/result.txt
14-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1/tools-default
15-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/metadata.log
16-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/user-benchmark-summary-debug.json
17-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/user-benchmark-summary.json
18-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/user-benchmark.cmd
6+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00
7+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/1
8+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/1/reference-result
9+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/1/sample1
10+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/1/sample1/result.txt
11+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/1/sample1/tools-default
12+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/metadata.log
13+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/user-benchmark-summary-debug.json
14+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/user-benchmark-summary.json
15+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/user-benchmark.cmd
1916
/var/tmp/pbench-test-bench/pbench-agent/pbench.log
2017
/var/tmp/pbench-test-bench/pbench-agent/tmp
21-
/var/tmp/pbench-test-bench/pbench-agent/tmp/pbench-user-benchmark__1900.01.01T00.00.00.iterations
18+
/var/tmp/pbench-test-bench/pbench-agent/tmp/pbench-user-benchmark_test-09_1900.01.01T00.00.00.iterations
2219
/var/tmp/pbench-test-bench/pbench-agent/tools-default
2320
/var/tmp/pbench-test-bench/pbench-agent/tools-default/mpstat
2421
/var/tmp/pbench-test-bench/pbench-agent/tools-default/sar
@@ -27,11 +24,28 @@ Running user-benchmark-script no-file
2724
/var/tmp/pbench-test-bench/pbench-agent/pbench.log:[info][1900-01-01T00:00:00.000000] [pbench-user-benchmark] Running user-benchmark-script no-file
2825
--- pbench.log file contents
2926
+++ test-execution.log file contents
30-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-collect-sysinfo --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00 --sysinfo=default end
27+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-collect-sysinfo --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00 --sysinfo=default end
3128
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-collect-sysinfo --sysinfo=default --check
32-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-metadata-log --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00 beg
33-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-metadata-log --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00 end
34-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-postprocess-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1
35-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-start-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1
36-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-stop-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1
29+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-metadata-log --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00 beg
30+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-metadata-log --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00 end
31+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-postprocess-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/1/sample1
32+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-start-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/1/sample1
33+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-stop-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/1/sample1
3734
--- test-execution.log file contents
35+
+++ pbench-user-benchmark_test-09_1900.01.01T00.00.00/metadata.log file contents
36+
[pbench]
37+
38+
[iterations/1]
39+
iteration_name = 1
40+
user_script = user-benchmark-script
41+
42+
--- pbench-user-benchmark_test-09_1900.01.01T00.00.00/metadata.log file contents
43+
+++ pbench-user-benchmark_test-09_1900.01.01T00.00.00/user-benchmark-summary.json file contents
44+
{
45+
"duration":0,
46+
"duration_units":"sec"
47+
}
48+
--- pbench-user-benchmark_test-09_1900.01.01T00.00.00/user-benchmark-summary.json file contents
49+
+++ pbench-user-benchmark_test-09_1900.01.01T00.00.00/user-benchmark.cmd file contents
50+
user-benchmark-script no-file 2>&1 | tee /var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-09_1900.01.01T00.00.00/1/sample1/result.txt
51+
--- pbench-user-benchmark_test-09_1900.01.01T00.00.00/user-benchmark.cmd file contents

‎agent/bench-scripts/gold/pbench-user-benchmark/test-10.txt‎

Lines changed: 37 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,21 @@
11
+++ Running test-10 pbench-user-benchmark
22
Running user-benchmark-script no-duration
33
--- Finished test-10 pbench-user-benchmark (status=0)
4-
{
5-
"duration":0,
6-
"duration_units":"sec",
7-
"that":"tother",
8-
"this":"that"
9-
}+++ pbench tree state
4+
+++ pbench tree state
105
/var/tmp/pbench-test-bench/pbench-agent
11-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00
12-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1
13-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/reference-result
14-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1
15-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1/result.txt
16-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1/tools-default
17-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/metadata.log
18-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/user-benchmark-summary-debug.json
19-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/user-benchmark-summary.json
20-
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/user-benchmark.cmd
6+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00
7+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/1
8+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/1/reference-result
9+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/1/sample1
10+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/1/sample1/result.txt
11+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/1/sample1/tools-default
12+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/metadata.log
13+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/user-benchmark-summary-debug.json
14+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/user-benchmark-summary.json
15+
/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/user-benchmark.cmd
2116
/var/tmp/pbench-test-bench/pbench-agent/pbench.log
2217
/var/tmp/pbench-test-bench/pbench-agent/tmp
23-
/var/tmp/pbench-test-bench/pbench-agent/tmp/pbench-user-benchmark__1900.01.01T00.00.00.iterations
18+
/var/tmp/pbench-test-bench/pbench-agent/tmp/pbench-user-benchmark_test-10_1900.01.01T00.00.00.iterations
2419
/var/tmp/pbench-test-bench/pbench-agent/tools-default
2520
/var/tmp/pbench-test-bench/pbench-agent/tools-default/mpstat
2621
/var/tmp/pbench-test-bench/pbench-agent/tools-default/sar
@@ -29,11 +24,30 @@ Running user-benchmark-script no-duration
2924
/var/tmp/pbench-test-bench/pbench-agent/pbench.log:[info][1900-01-01T00:00:00.000000] [pbench-user-benchmark] Running user-benchmark-script no-duration
3025
--- pbench.log file contents
3126
+++ test-execution.log file contents
32-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-collect-sysinfo --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00 --sysinfo=default end
27+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-collect-sysinfo --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00 --sysinfo=default end
3328
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-collect-sysinfo --sysinfo=default --check
34-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-metadata-log --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00 beg
35-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-metadata-log --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00 end
36-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-postprocess-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1
37-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-start-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1
38-
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-stop-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark__1900.01.01T00.00.00/1/sample1
29+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-metadata-log --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00 beg
30+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-metadata-log --group=default --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00 end
31+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-postprocess-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/1/sample1
32+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-start-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/1/sample1
33+
/var/tmp/pbench-test-bench/test-execution.log:/var/tmp/pbench-test-bench/opt/pbench-agent/unittest-scripts/pbench-stop-tools --group=default --iteration=1 --dir=/var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/1/sample1
3934
--- test-execution.log file contents
35+
+++ pbench-user-benchmark_test-10_1900.01.01T00.00.00/metadata.log file contents
36+
[pbench]
37+
38+
[iterations/1]
39+
iteration_name = 1
40+
user_script = user-benchmark-script
41+
42+
--- pbench-user-benchmark_test-10_1900.01.01T00.00.00/metadata.log file contents
43+
+++ pbench-user-benchmark_test-10_1900.01.01T00.00.00/user-benchmark-summary.json file contents
44+
{
45+
"duration":0,
46+
"duration_units":"sec",
47+
"that":"tother",
48+
"this":"that"
49+
}
50+
--- pbench-user-benchmark_test-10_1900.01.01T00.00.00/user-benchmark-summary.json file contents
51+
+++ pbench-user-benchmark_test-10_1900.01.01T00.00.00/user-benchmark.cmd file contents
52+
user-benchmark-script no-duration 2>&1 | tee /var/tmp/pbench-test-bench/pbench-agent/pbench-user-benchmark_test-10_1900.01.01T00.00.00/1/sample1/result.txt
53+
--- pbench-user-benchmark_test-10_1900.01.01T00.00.00/user-benchmark.cmd file contents

0 commit comments

Comments
 (0)