Discussion:
Test design issue.
(too old to reply)
Rob Landley
2017-04-24 22:18:32 UTC
Permalink
Raw Message
I want an ls.test checking "ls -b" via some variant of:

mkdir blah
touch blah/"$(echo -e "$(X=0;while [ $X -lt 255 ];do X=$(($X+1));[ $X
-eq 47 ]&& continue;printf '\\x%02x' $X; done)")"
ls -b blah

And then make sure it's escaping everything it should, but the ubuntu
14.04 ls -b and toybox's ls -b have one difference: for ascii 27 (the
escape char) ubuntu says "\033" and toybox says "\e".

I guess I can break it into 3 tests with the middle one SKIP_HOST? Seems
awkward...

Rob
enh
2017-04-25 17:28:39 UTC
Permalink
Raw Message
note that the existing SKIP_HOST mechanism is very error prone. iirc none
of the sed tests actually run because one near the top needs SKIP_HOST and
it never gets unset.

SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
distinguish "host tool is just plain broken" from "we deliberately chose to
diverge [because ...]".

cases like sh.test are awkward too... exponentiation is a bash extension to
the more widespread arithmetic extension (which is all the test tests). so
although it's "sh.test", it explicitly invokes "bash". and if you "fix" it
by running sh, bash-as-sh fails too. for now i just have an explicit case
to skip sh.test in my Android toybox test runner, but in an ideal world it
would be more fine-grained than that.

here though, is there a strong reason to prefer \e? otherwise why not just
do what the host does in this case?
Post by Rob Landley
mkdir blah
touch blah/"$(echo -e "$(X=0;while [ $X -lt 255 ];do X=$(($X+1));[ $X
-eq 47 ]&& continue;printf '\\x%02x' $X; done)")"
ls -b blah
And then make sure it's escaping everything it should, but the ubuntu
14.04 ls -b and toybox's ls -b have one difference: for ascii 27 (the
escape char) ubuntu says "\033" and toybox says "\e".
I guess I can break it into 3 tests with the middle one SKIP_HOST? Seems
awkward...
Rob
_______________________________________________
Toybox mailing list
http://lists.landley.net/listinfo.cgi/toybox-landley.net
--
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
Rob Landley
2017-04-26 21:41:29 UTC
Permalink
Raw Message
Post by enh
note that the existing SKIP_HOST mechanism is very error prone. iirc
none of the sed tests actually run because one near the top needs
SKIP_HOST and it never gets unset.
I thought that "VARIABLE=value command" worked the same for shell
functions as it does for external commands? Hmmm...

$ make test_sed > one.txt
$ TEST_HOST=1 make test_sed > two.txt
$ diff -u one.txt two.txt | grep '^[+-]'
...
-PASS: sed sed - - twice
-PASS: sed match \n literal n
-PASS: sed c empty continuation
-PASS: sed 'y/a\bc/de\f/'
+FAIL: sed 'y/a\bc/de\f/'
-PASS: sed skip start of range
+FAIL: sed skip start of range

The first three are the ones we skip, the second the host version is
failing but the target version isn't so possibly they should be
annotated too. (Or at least have a comment...)

What shell are you using to run this? (Not bash?)
Post by enh
SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
distinguish "host tool is just plain broken" from "we deliberately chose
to diverge [because ...]".
Which is why I added comments before them, ala:

# This segfaults ubuntu 12.04's sed. No really.
SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"

But I see I haven't kept up with that. Blah. (Presumably git annotate
should give a commit comment, and/or let me know what date to look for a
mailing list entry or blog post. I'll throw it on the todo list.)
Post by enh
cases like sh.test are awkward too...
Yeah shellit() tries to compensate a bit but I haven't opened the toysh
can of worms yet. (Tim Bird asked me what it would take to do toysh, and
my estimate was 6 months full-time devoted to it, but he hasn't got a
budget to buy my time from my employer. Meanwhile $DAYJOB is focused on
winning solar/wind grid integration monitoring contracts. The market
opportunity for that is likely to explode in about 3 years and we need
to deploy stuff people can buy before then and be ready to scale up, so
getting any time to spend on toybox takes a bit of pushing...)

On the bright side I got https://github.com/landley/mkroot a bit more
fluffed out this week. Once I get all the musl-cross-make targets
booting, I can coordinate with Rich to get him to tag and ship an actual
release with binaries you can download (maybe he'll even put the
mcm-buildall.sh script in _his_ repo instead of mine).

Building kernels with mkroot needs toybox commit 9be140ebb29f so I'll
have to cut a new release here to coincide with the mkroot release, but
it's coming up on time for that anyway...
Post by enh
exponentiation is a bash extension
to the more widespread arithmetic extension (which is all the test
tests).
Oh toysh is going to need quite possibly a thousand lines of tests. The
tests I have _now_ are just ones that I had lying around because I was
poking at something that was doing this...

I thought it was bc/dc but:

$ dc "1+2"
dc: Could not open file 1+2
$ bc "1+2"
File 1+2 is unavailable.

Those are both _way_ too stupid to use as a quick and dirty calculator
from the command line. (I usually do echo $((1+2)) when I don't need
floating point, and fire up python when I do. But there should be a
better way, possibly I should have toysh $(()) support floating point?
Hmmm...)
Post by enh
so although it's "sh.test", it explicitly invokes "bash".
Because toysh should provide the bash alias along with bash functionality.

Ubuntu's epically stupid https://wiki.ubuntu.com/DashAsBinSh broke the
#!/bin/sh symlink to point to a broken shell (seriously, at the time it
segfaulted and got signal handling wrong and...), so scripts explicitly
say #!/bin/bash instead. I intend to provide a reasonable bash
replacement that runs those scripts.

Bash was the first program linux ever ran (and running bash was the
reason Linus implemented system calls in his term program), so it was
the standard shell of Linux from 1991 to 2006. Ubuntu's stated reason
for changing this (speeding up init scripts) didn't work (they then
implemented upstart to get parallelism), but they never admitted the
mistake and reverted it. They said that changing the #! at the start of
each init script was too intrusive a change, so they broke the kernel
build instead.

That's why all my scripts say #!/bin/bash and not #!/bin/sh. I need to
replace _that_.
Post by enh
and if you "fix" it by running sh, bash-as-sh fails too. for now i just have an
explicit case to skip sh.test in my Android toybox test runner, but in
an ideal world it would be more fine-grained than that.
My infrastructure checks .config and should skip tests for commands that
are disabled?

Alas, I don't have a "pending" for the tests directory. In an ideal
world I'd be making progress fast enough this wouldn't be an issue. In
practice we're coming up on the one year anniversary of $DAYJOB's
"funding knothole" where we've been in perpetual crisis mode ever since.
(It's compelling technology I really want to see ship, but the pay sucks
and it's eating my life. Doing what I can here...)

In the meantime, I can add a check for "bash" in the $PATH and skip if
not". Does that help?
Post by enh
here though, is there a strong reason to prefer \e? otherwise why not
just do what the host does in this case?
Because "man echo" says \e and my escape/unescape plumbing is
symmetrical. I'd have to special case it to _not_ understand this.

I suppose I can also have the test run the output through sed to replace
"\e" with "\033" and then it'd except either?

Rob
enh
2017-04-26 21:56:19 UTC
Permalink
Raw Message
Post by Rob Landley
Post by enh
note that the existing SKIP_HOST mechanism is very error prone. iirc
none of the sed tests actually run because one near the top needs
SKIP_HOST and it never gets unset.
I thought that "VARIABLE=value command" worked the same for shell
functions as it does for external commands? Hmmm...
$ make test_sed > one.txt
$ TEST_HOST=1 make test_sed > two.txt
$ diff -u one.txt two.txt | grep '^[+-]'
...
-PASS: sed sed - - twice
-PASS: sed match \n literal n
-PASS: sed c empty continuation
-PASS: sed 'y/a\bc/de\f/'
+FAIL: sed 'y/a\bc/de\f/'
-PASS: sed skip start of range
+FAIL: sed skip start of range
The first three are the ones we skip, the second the host version is
failing but the target version isn't so possibly they should be
annotated too. (Or at least have a comment...)
What shell are you using to run this? (Not bash?)
hmm, maybe this got fixed since i last saw it? with TEST_HOST=1 make
test_sed i'm not seeing any "SKIP" output, whereas when i ran in to this i
saw a whole wall of yellow "SKIP" lines.
Post by Rob Landley
Post by enh
SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
distinguish "host tool is just plain broken" from "we deliberately chose
to diverge [because ...]".
# This segfaults ubuntu 12.04's sed. No really.
SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"
But I see I haven't kept up with that. Blah. (Presumably git annotate
should give a commit comment, and/or let me know what date to look for a
mailing list entry or blog post. I'll throw it on the todo list.)
exactly: having the skip mechanism require a string would help keep us
honest.
Post by Rob Landley
Post by enh
cases like sh.test are awkward too...
Yeah shellit() tries to compensate a bit but I haven't opened the toysh
can of worms yet. (Tim Bird asked me what it would take to do toysh, and
my estimate was 6 months full-time devoted to it, but he hasn't got a
budget to buy my time from my employer. Meanwhile $DAYJOB is focused on
winning solar/wind grid integration monitoring contracts. The market
opportunity for that is likely to explode in about 3 years and we need
to deploy stuff people can buy before then and be ready to scale up, so
getting any time to spend on toybox takes a bit of pushing...)
On the bright side I got https://github.com/landley/mkroot a bit more
fluffed out this week. Once I get all the musl-cross-make targets
booting, I can coordinate with Rich to get him to tag and ship an actual
release with binaries you can download (maybe he'll even put the
mcm-buildall.sh script in _his_ repo instead of mine).
Building kernels with mkroot needs toybox commit 9be140ebb29f so I'll
have to cut a new release here to coincide with the mkroot release, but
it's coming up on time for that anyway...
Post by enh
exponentiation is a bash extension
to the more widespread arithmetic extension (which is all the test
tests).
Oh toysh is going to need quite possibly a thousand lines of tests. The
tests I have _now_ are just ones that I had lying around because I was
poking at something that was doing this...
mksh has a thousand line perl script to run 13k lines of tests that look
like

name: xtrace-2
description:
Check that "set -x" is off during PS4 expansion
stdin:
f() {
print -n "(f1:$-)"
set -x
print -n "(f2:$-)"
}
PS4='[(p:$-)$(f)] '
print "(o0:$-)"
set -x -o inherit-xtrace
print "(o1:$-)"
set +x
print "(o2:$-)"
expected-stdout:
(o0:sh)
(o1:shx)
(o2:sh)
expected-stderr:
[(p:sh)(f1:sh)(f2:sh)] print '(o1:shx)'
[(p:sh)(f1:sh)(f2:sh)] set +x
Post by Rob Landley
$ dc "1+2"
dc: Could not open file 1+2
$ bc "1+2"
File 1+2 is unavailable.
Those are both _way_ too stupid to use as a quick and dirty calculator
from the command line. (I usually do echo $((1+2)) when I don't need
floating point, and fire up python when I do. But there should be a
better way, possibly I should have toysh $(()) support floating point?
Hmmm...)
Post by enh
so although it's "sh.test", it explicitly invokes "bash".
Because toysh should provide the bash alias along with bash functionality.
Ubuntu's epically stupid https://wiki.ubuntu.com/DashAsBinSh broke the
#!/bin/sh symlink to point to a broken shell (seriously, at the time it
segfaulted and got signal handling wrong and...), so scripts explicitly
say #!/bin/bash instead. I intend to provide a reasonable bash
replacement that runs those scripts.
Bash was the first program linux ever ran (and running bash was the
reason Linus implemented system calls in his term program), so it was
the standard shell of Linux from 1991 to 2006. Ubuntu's stated reason
for changing this (speeding up init scripts) didn't work (they then
implemented upstart to get parallelism), but they never admitted the
mistake and reverted it. They said that changing the #! at the start of
each init script was too intrusive a change, so they broke the kernel
build instead.
That's why all my scripts say #!/bin/bash and not #!/bin/sh. I need to
replace _that_.
Post by enh
and if you "fix" it by running sh, bash-as-sh fails too. for now i just
have an
Post by enh
explicit case to skip sh.test in my Android toybox test runner, but in
an ideal world it would be more fine-grained than that.
My infrastructure checks .config and should skip tests for commands that
are disabled?
huh. i hadn't thought of that (though i had thought of its moral
equivalent: checking to see whether the tool in question is a symlink to
`toybox`). that's an option, though i'm potentially still interested in
running the toybox tests against a non-toybox implementation. (but maybe my
runner should check against .config and explicitly say when we're not
running toybox.)
Post by Rob Landley
Alas, I don't have a "pending" for the tests directory. In an ideal
world I'd be making progress fast enough this wouldn't be an issue. In
practice we're coming up on the one year anniversary of $DAYJOB's
"funding knothole" where we've been in perpetual crisis mode ever since.
(It's compelling technology I really want to see ship, but the pay sucks
and it's eating my life. Doing what I can here...)
In the meantime, I can add a check for "bash" in the $PATH and skip if
not". Does that help?
if you hadn't committed it already i'd say "don't worry about it": there's
not enough in sh.test at the moment to worry about. what i _ought_ to be
doing for shell testing is building a perl binary to run the mksh suite,
but the thought of building perl in 2017 really stretches my pragmatism to
its limit!

i'm more interested in seeing the gzip/gunzip/zcat tests go in so we can
start arguing about how to upstream my toybox rewrite of the toolbox
zlib-based gzip/gunzip/zcat. (i did the port on monday, so i do have a
toybox gzip/gunzip/zcat that passes the tests, but i didn't even try to
integrate the half-finished gz code from toys/pending/compress.c.)
Post by Rob Landley
Post by enh
here though, is there a strong reason to prefer \e? otherwise why not
just do what the host does in this case?
Because "man echo" says \e and my escape/unescape plumbing is
symmetrical. I'd have to special case it to _not_ understand this.
I suppose I can also have the test run the output through sed to replace
"\e" with "\033" and then it'd except either?
Rob
_______________________________________________
Toybox mailing list
http://lists.landley.net/listinfo.cgi/toybox-landley.net
--
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
Rob Landley
2017-04-27 06:14:00 UTC
Permalink
Raw Message
Post by Rob Landley
Post by enh
note that the existing SKIP_HOST mechanism is very error prone. iirc
none of the sed tests actually run because one near the top needs
SKIP_HOST and it never gets unset.
I thought that "VARIABLE=value command" worked the same for shell
functions as it does for external commands? Hmmm...
$ make test_sed > one.txt
$ TEST_HOST=1 make test_sed > two.txt
$ diff -u one.txt two.txt | grep '^[+-]'
...
-PASS: sed sed - - twice
-PASS: sed match \n literal n
-PASS: sed c empty continuation
-PASS: sed 'y/a\bc/de\f/'
+FAIL: sed 'y/a\bc/de\f/'
-PASS: sed skip start of range
+FAIL: sed skip start of range
The first three are the ones we skip, the second the host version is
failing but the target version isn't so possibly they should be
annotated too. (Or at least have a comment...)
What shell are you using to run this? (Not bash?)
hmm, maybe this got fixed since i last saw it?
Not that I recall, and commit log of sed.test isn't saying so either.

with TEST_HOST=1 make
Post by Rob Landley
test_sed i'm not seeing any "SKIP" output, whereas when i ran in to this
i saw a whole wall of yellow "SKIP" lines.
Hmmm... I just tried "BLAH=42 echo hello; echo $BLAH" on the android M
shell on my phone and it didn't persistently set it, but I don't have an
O test environment lying around... (I really need to work out how to
build individual AOSP packages outside of AOSP. I'd like a static
version of mksh and such to compare things against. And a bionic
toolchain I can build toybox against on an ubuntu host...)

I don't recall fixing this here, and am not seeing anything obvious in
the logs, but that doesn't mean much. Remember my recent "I should teach
tests to ignore whitespace" and fixing ls -m output to space
differently? A year ago I did:

commit fcbf55ad565861e22e1cb297e6954b9d82f5094b
Author: Rob Landley <***@landley.net>
Date: Thu Jun 30 10:39:23 2016 -0500

Add NOSPACE=1 to allow tests to pass with diff -b

Which is in news.html but not code.html. I should add a section
documenting the test suite...

One of the reasons there's a whole lotta "squirrel!" in my development
process is when I get a reproducible bug I tend to _stop_ and chase that
sucker. Reproducible bugs are gold, and tend to submerge again if you
let the environment they reproduce in drift. Submerge is NOT the same as
get fixed, it's like roaches. You see it, you kill it, then you spray
for it. Do not let it breed. So fixing reproducible bugs can take
priority of implementing features, even halfway through...
Post by Rob Landley
Post by enh
SKIP_HOST is also not very intention-revealing. i'd prefer to explicitly
distinguish "host tool is just plain broken" from "we deliberately chose
to diverge [because ...]".
# This segfaults ubuntu 12.04's sed. No really.
SKIP_HOST=1 testing 'sed - - twice' 'sed "" - -' "hello\n" "" "hello\n"
But I see I haven't kept up with that. Blah. (Presumably git annotate
should give a commit comment, and/or let me know what date to look for a
mailing list entry or blog post. I'll throw it on the todo list.)
exactly: having the skip mechanism require a string would help keep us
honest.
In theory you could do SKIP_HOST="this is why" now, it only cares that
it's not an empty string. It'd be a bit _more_ error prone though
because you'd have to do:

SKIP_HOST="Tea not hot enough to drive atomic vector plotter" \
testing "improbability drive" "whale" "petunias" "" ""

And if you miss the \ at the end of the line you get the trailing
SKIP_HOST again. (That said, you'd presumably _notice_ that rather a
lot. But only if you remember to TEST_HOST as well as target. :)

If I get the test suite in a reasonable shape, I can add a line to my
release.txt checklist, which is currently:

news.html - release notes
Update version in main.c
- git commit main.c www/news.html
tag repo
- git tag $VER
- git push --tags
source tarball
git archive --prefix=toybox-$VER/ $VER | gzip -9 > toybox-$VER.tar.gz
scp toybox-$VER.tar.gz landley.net:
scp www/news.html landley.net:
binaries
cd ../clean
git pull ../toy3
../relbuild.sh
update help.html
./toybox help -ah > www/help.html
scp www/help.html landley.net:landley.net/toybox/
scripts/mkstatus.py -> status.html
- scp www/status.gen landley.net:landley.net/toybox/
roadmap.html
code.html
Inform mailing list

This _used_ to have "build linux from scratch under aboriginal linux" on
all targets, but aboriginal fell by the wayside because
(http://lists.landley.net/pipermail/aboriginal-landley.net/2016-May/002567.html)
and I'm finally doing mkroot to replace it now...

(You may have noticed I'm a big believer in testing with real-world data. :)
Post by Rob Landley
Oh toysh is going to need quite possibly a thousand lines of tests. The
tests I have _now_ are just ones that I had lying around because I was
poking at something that was doing this...
mksh has a thousand line perl script to run 13k lines of tests that look
like
I am totally stealing that if you'll let me. :)
Post by Rob Landley
name: xtrace-2
Check that "set -x" is off during PS4 expansion
f() {
print -n "(f1:$-)"
set -x
print -n "(f2:$-)"
}
PS4='[(p:$-)$(f)] '
print "(o0:$-)"
set -x -o inherit-xtrace
print "(o1:$-)"
set +x
print "(o2:$-)"
(o0:sh)
(o1:shx)
(o2:sh)
[(p:sh)(f1:sh)(f2:sh)] print '(o1:shx)'
[(p:sh)(f1:sh)(f2:sh)] set +x
Given the difference in file formats (mine has a lot of \n in it), i
expect my ballkpark of a thousand tests and your 13k lines may be a
similar amount of tests. :)
Post by Rob Landley
Post by enh
and if you "fix" it by running sh, bash-as-sh fails too. for now i just have an
explicit case to skip sh.test in my Android toybox test runner, but in
an ideal world it would be more fine-grained than that.
My infrastructure checks .config and should skip tests for commands that
are disabled?
huh. i hadn't thought of that (though i had thought of its moral
equivalent: checking to see whether the tool in question is a symlink to
`toybox`). that's an option, though i'm potentially still interested in
running the toybox tests against a non-toybox implementation. (but maybe
my runner should check against .config and explicitly say when we're not
running toybox.)
I think TEST_HOST skips the .config check already. (Haven't looked in a
while.)

Properly doing the test suite has been on my todo list for a while as
one of the Big Projects between here and the 1.0 release. The tests I
have there are the tests I ran during development (or when fixing a bug)
that I want to memorialize, but that's not the same as scrutinizing the
spec, man page, help text, and code to come up with full coverage.
Post by Rob Landley
In the meantime, I can add a check for "bash" in the $PATH and skip if
not". Does that help?
there's not enough in sh.test at the moment to worry about. what i
_ought_ to be doing for shell testing is building a perl binary to run
the mksh suite, but the thought of building perl in 2017 really
stretches my pragmatism to its limit!
Ouch. I've done that
(https://github.com/landley/control-images/blob/master/images/lfs-bootstrap/mnt/build/perl.sh)
but it's way stale and kinda sad and I've spent a lot more time porting
things _off_ of perl (such as the perl removal work I did for the linux
kernel build).

Point me at your shell tests and maybe I can port them to something that
doesn't require perl. I need proper toybox shell tests anyway, and
making toysh pass the test suite you're already using would be an
important step in implementing something you could potentially move to
in a future release. (It's almost a hard prerequisite...)
Post by Rob Landley
i'm more interested in seeing the gzip/gunzip/zcat tests go in so we can
start arguing about how to upstream my toybox rewrite of the toolbox
zlib-based gzip/gunzip/zcat. (i did the port on monday, so i do have a
toybox gzip/gunzip/zcat that passes the tests, but i didn't even try to
integrate the half-finished gz code from toys/pending/compress.c.)
Eh, send me what you've got and I can do that. I'm going to try to
devote this weekend to it.

I _did_ have gunzip and zcat working fine at one point, I just screwed
it up since. (Somebody actually sent a bug report with fix to github,
but it was the wrong fix and I had a window open and my netbook battery
died and the todo item got shuffled down the list in the reboot because
"open window" trumps "need to read one of my ~5 active todo.txt files".)
Post by Rob Landley
Post by enh
here though, is there a strong reason to prefer \e? otherwise why not
just do what the host does in this case?
Because "man echo" says \e and my escape/unescape plumbing is
symmetrical. I'd have to special case it to _not_ understand this.
I suppose I can also have the test run the output through sed to replace
"\e" with "\033" and then it'd except either?
This todo item meanwhile is recorded via a local modification to my
tests/ls.test adding two comment lines, which aren't checked in but
which show up in "git diff". (I also added a line to code.html about the
test suite and NOSPACE=1, same trick.)

It's not so bad if I keep on top of them. And if I don't, the
mechanism(s) by which I record them probably don't help. :P

Rob

Loading...