2016-03-19 01:24:50 UTC
parts are ready to merge.
I think we should change the test "protocol" to be a subprocess rather
sourcing the .test file, as you alluded to. But that can be done in a
separate step on top of this.
There's also some cruft like '. testing.sh' at the top of most tests,
but it doesn't seem to exist. As long as I'm going in the right
direction I can fix all this.
Summary after patch 1 but before patch 2:
Tested 66 toybox commands: 90 test failures
Commands with test failures: bzcat chattr date find groupadd groupdel
hostname ls modinfo mount mv pgrep pkill renice tar touch xzcat zcat
(skipped commands not listed)
After patch 2:
Tested 61 toybox commands: 36 test failures
Commands with test failures: date find ls modinfo mv pgrep pkill
renice tar zcat
Commands skipped: chattr chgrp chown groupadd groupdel hostname
ifconfig losetup mount useradd
make: *** [tests] Error 1
[PATCH 1/2] Refactor the test harness to triage tests, and fix some
- Fix a name clash bug where 'make test' would run all tests and then try to
build the 'test' binary. 'make test_bin' is special cased for now.
- Make the test targets .PHONY, since we're not creating files called test_sed,
- Use here docs to write Makefile fragments, for readability and to be
consistent with the rest of the script. Refactor into functions.
- Add comments.
- Add the output binary name as the first param. This fixes an issue where
'make; make test_sed' would leave you without a 'toybox' binary because
single.sh created another toybox binary and renamed it to sed.
- Changed the 'testing' function to return 0. This fixes a bug where the
return value of the script depends on the return code of the last command --
NOT whether it passed or failed. For example, 'make test_test' would fail
with code 1, even though all tests passed (because 'test' exits 1 validly).
On the other hand, tests with failures would still return 0 if the last
command didn't fail.
- Use local variables instead of globals since we are sharing globals with the
- Add a common function to skip tests if not running as root, and keep track of
- Refactor it into separate functions/actions and add usage.
- Fix a bug where tests would succeed under 'make test_$COMMAND' but fail under
'make test'. The problem is if you are running the xzcat command, 'tar'
could be either the host binary or the toybox binary, depending on what tools
are in the modified test $PATH. To fix this, the test harness finds the host
binary and sets $HOST_BIN_TAR, etc. before invoking the test.
- Keep track of the number of failures per *command*, and exit 1 if any failed
- Use a distinct test directory for each command, instead of repeatedly setting
up and tearing down the same one.
- When running all tests, print out a summary at the end.
- Call scripts/make.sh directly instead of 'make'.
- Add comments.
Makefile | 2 +-
scripts/genconfig.sh | 103 ++++++++++++++++++++++++++---------
scripts/make.sh | 16 ++++--
scripts/runtest.sh | 40 +++++++++-----
scripts/single.sh | 28 +++++++---
scripts/test.sh | 149 +++++++++++++++++++++++++++++++++++++++++----------
6 files changed, 258 insertions(+), 80 deletions(-)
[PATCH 2/2] Various test fixes and cleanup.
bzcat: Fix a test that was succeeding for the wrong reason. The test
was looking for a 1 exit code in the case of bad data, but there was a
typo in the input path, so it failed 1 for the wrong reason.
bzcat, xzcat, zcat: Use $HOST_BIN_TAR to fix the issue where they fail
under 'make test' but not 'make test_bzcat'.
hostname: Use $HOST_BIN_HOSTNAME, and skip one test if not root.
chgrp, groupadd, groupdel, ifconfig, losetup, mount, useradd: Skip if
pgrep, pkill: style cleanup and speedup of the tests (change sleep 1 to
sleep 0.1). No fixes yet.
renice: Fix some quoting problems (revealing further errors, not fixed)
tar: Style cleanup in preparation to fix failures under 'make test' but
not 'make test_tar' (due to host/toybox discrepancy)
touch: Use $HOST_BIN_DATE because toybox doesn't support %N nanoseconds
tests/bzcat.test | 41 ++++++++++---------
tests/chattr.test | 4 +-
tests/chgrp.test | 7 +---
tests/chown.test | 7 +---
tests/groupadd.test | 2 +
tests/groupdel.test | 2 +
tests/hostname.test | 16 +++++---
tests/ifconfig.test | 7 +---
tests/losetup.test | 7 +---
tests/mount.test | 6 ++-
tests/pgrep.test | 51 ++++++++++++------------
tests/pkill.test | 54 ++++++++++++-------------
tests/renice.test | 8 ++--
tests/tar.test | 111 ++++++++++++++++++++++++++++++++--------------------
tests/touch.test | 6 +--
tests/useradd.test | 7 +---
tests/xzcat.test | 36 +++++++++--------
tests/zcat.test | 39 +++++++++---------
18 files changed, 216 insertions(+), 195 deletions(-)