Working with tests

Introduction

Tests come in two broad categories, unit tests and integration tests.

The unit tests are faster, and pinpoint errors more precisely because they are tailored to the detailed knowledge of how the code works. Integration tests check the behavior and generally don't care how something is coded as long as you get the desired result. In this regard they are more important because they check more what you care about. But they are slower and when something fails, the source of the error is less precise.

Unit tests

Unit tests are located in the directory 'test/unit'. From the top 'kshdb' directory you can run all of the tests via:

 make test-unit

Or you can go into the directory test/unit and run:

make test

Unit tests use Kate Ward's shunit2 script which is located inside the directory.

If a unit test fails, you may get something like this:

make test 
/bin/ksh93 github/kshdb/test/unit/shunit2 test-alias.sh test-break.sh test-cmd-complete.sh test-columns.sh test-dbg-opts.sh test-eval.sh test-examine.sh test-file.sh test-filecache.sh test-frame.sh test-journal.sh test-lib-complete.sh test-msg.sh test-pre.sh test-run.sh test-save-restore.sh test-tty.sh
test_alias
ASSERT:expected:<up2> but was:<up>

The above means that function test_alias() failed. It happens to be located in file test-alias.sh which is suggested by the name, but also by its position, first, in the shunit2 invocation.

You can run that test individually:

cd test/unit
 ./test_alias
test_alias
ASSERT:expected:<up2> but was:<up>
Ran 1 test. 7 assert(s): 6 passed, 1 failed, 0 skipped.
FAILED (failures=1)

If one of the unit tests fail, we don't even try to run integration tests which are described next.

Integration tests

The integration tests reside in directory test/integration. From the top 'kshdb' directory you can run all of the tests via:

 make test-integration

Or you can go into the directory test/integration and run:

make test

If an integration test fails, you may get something like this:

$ make test
make  check-common.sh
make[1]: Entering directory `kshdb/test/integration'
make[1]: `check-common.sh' is up to date.
make[1]: Leaving directory `kshdb/test/integration'
make  check-TESTS
make[1]: Entering directory `kshdb/test/integration
make[2]: Entering directory `kshdb/test/integration
PASS: test-break
FAIL: test-bug-args
...
============================================================================
Testsuite summary for kshdb 0.07dev
============================================================================
# TOTAL: 17
# PASS:  8
# SKIP:  0
# XFAIL: 0
# FAIL:  9
# XPASS: 0
# ERROR: 0

The above shows that file bug-args failed (among others). You can run that test individually:

cd test/integration
 ./test_bug_args
./test-bug-args
-- kshdb/test/integration/bug-args.check      2014-12-29 16:08:01.514200721 -0500
+++ kshdb/test/data/bug-args.right     2011-04-17 17:26:47.000000000 -0400
@@ -2,21 +2,29 @@
 echo First parm is:
 +# Debugger test to see that parameter handling of $1, $2, etc is correct.
 +print $#
+0
 +print $5
+
+step 2
 First parm is:
 (bug-args.sh:4):
 shift 2

The above gives diff -u output between expeced results kshdb/test/data/bug-args.right and the actual output when run kshdb/test/integration/bug-args.check. If you prefer you can use your own file comparitor routine to compare these files since they stay in the filesystem when there is an error. Going with the diff output above though, we see that there were some expected lines that we didn't get. Namely:

0

and a blank the blank line after "+print 5". And this is the bug here, the print routine is not printing

Diff output adds a "+" to lines that are added and a "-" to lines that were deleted and "!" to lines that were modified. If there was no change there is a space.

The output above may be a little confusing because the program output also has a "+" in column 1. Again, the diff output everything is shifted over by one column so you will see " +" or " +step2" where the actual ouput was "+" or "+step2" without the leading space.