Discussion:
[PATCH] Add the gzip/gunzip/zcat tests I wrote for toolbox gzip/gunzip/zcat.
(too old to reply)
enh
2017-04-24 20:43:51 UTC
Permalink
Raw Message
Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
another day, but at least the tests are easy...

(These tests pass with TEST_HOST and on the toolbox versions, but the
toybox toys are in pending and very broken.)
---
tests/gunzip.test | 50 +++++++++++++++++++++++++++++++++++
tests/gzip.test | 78
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/zcat.test | 34 ++++++++++--------------
3 files changed, 142 insertions(+), 20 deletions(-)
create mode 100644 tests/gunzip.test
create mode 100644 tests/gzip.test
mode change 100755 => 100644 tests/zcat.test
Rob Landley
2017-04-26 21:43:26 UTC
Permalink
Raw Message
Post by enh
Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
another day,
Blah, I should take this one.

I have deflate decompression side implemented already (and working at
one point, although I just tested "make zcat" and got a bad crc
decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
compression side implemented and somewhere around here is a version
that's about 2/3 through. I stopped at "where do dictionary resets
happen" and basically go the answer "nobody seems to agree, just do it
every 250k". (Which means my gzip and other gzip won't necessarily
produce the same binary, but it should extract the same. There's
actually a format hitch you can do sticking in gratuitous resets to
allow parallel decompression, it's a bit like mp4 streaming mode
sticking in extra keyframes, only noticeably cheaper. :)

Adding zlib mode to that is basically the same plumbing as
toys/*/md5sum.c does for libcrypto.

Lemme try to find a chunk of time to devote to that...
Post by enh
but at least the tests are easy...
(These tests pass with TEST_HOST and on the toolbox versions, but the
toybox toys are in pending and very broken.)
The problem is this is just testing that gzip and gunzip reverse each
other. If they share infrastructure and screw something up the same way,
what have you proved? Also you can't test gunzip without gzip being present.

For bzip I added a tests/files/bzcat (and already had a lot of
tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
files for reference under tests/files.

I'll try to take a closer look at this later...

Rob
enh
2017-04-26 22:03:56 UTC
Permalink
Raw Message
Post by Rob Landley
Post by enh
Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
another day,
Blah, I should take this one.
I have deflate decompression side implemented already (and working at
one point, although I just tested "make zcat" and got a bad crc
decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
compression side implemented and somewhere around here is a version
that's about 2/3 through. I stopped at "where do dictionary resets
happen" and basically go the answer "nobody seems to agree, just do it
every 250k". (Which means my gzip and other gzip won't necessarily
produce the same binary, but it should extract the same. There's
actually a format hitch you can do sticking in gratuitous resets to
allow parallel decompression, it's a bit like mp4 streaming mode
sticking in extra keyframes, only noticeably cheaper. :)
Adding zlib mode to that is basically the same plumbing as
toys/*/md5sum.c does for libcrypto.
Lemme try to find a chunk of time to devote to that...
if you're actually going to start to look, i'll attach my port of the
current toolbox implementation.

---
Config.in | 6 ++
scripts/make.sh | 2 +-
toys/pending/compress.c | 93 ----------------------
toys/pending/gzip.c | 199
++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 206 insertions(+), 94 deletions(-)
create mode 100644 toys/pending/gzip.c
Post by Rob Landley
Post by enh
but at least the tests are easy...
(These tests pass with TEST_HOST and on the toolbox versions, but the
toybox toys are in pending and very broken.)
The problem is this is just testing that gzip and gunzip reverse each
other. If they share infrastructure and screw something up the same way,
what have you proved? Also you can't test gunzip without gzip being present.
For bzip I added a tests/files/bzcat (and already had a lot of
tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
files for reference under tests/files.
I'll try to take a closer look at this later...
Rob
--
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
Rob Landley
2017-05-08 23:37:20 UTC
Permalink
Raw Message
Post by enh
if you're actually going to start to look, i'll attach my port of the
current toolbox implementation.
Ok, carving out a half hour to look at this... optargs should really
have [-123456789] at the end so "gzip -3 -9" is -9 but "gzip -9 -3" is
-3. While we're at it, OLDTOY() was there so these things could share
option strings, except why does zcat have -cd? (And the three main
functions... if oldtoy isn't handling this right it needs fixing. Do you
care about the standalone builds, by the way? that was always the
complicating part, chording together shared infrastructure...)

I need to spend a 3 day weekend fixing the help infrastructure so it can
do reasonable includes, the duplication of -c and -f help text here
pains me.

Ok, jumping down to main()... it seems like there should be a
first_bit() function in lib/lib.c. Throw it in locally and move it when
I find a second user. (Leaving it _not_ static to remind me about the
library collation next time I hunt around for non-static functions.)

Setting FLAG_c for no arguments can't be right, assuming gzip is doing
what I think it is...

gunzip one.gz - three.gz < two.gz

And yes two gets output to stdout. So gunzip - writes "back" to stdout
automatically, so the loopfiles() behavior of "no arguments means one
argument equivalent to -" is fine and the callback has to handle that.

loopfiles() second argument is a function pointer, you don't need a
do_gz() wrapper you can just have the test in loopfiles to pass in the
right argument. (It's not changing partway through the loop...)

Hmmm, there's plumbing in things like patch and sed -i using
copy_tempfile() to create a temporary file we write to (and delete if
we're interrupted, via an atexit() variant). You're leaving the
half-finished output file if we get interrupted, is that what we should do?

Also, some commands go to a bit of trouble _not_ to exit immediately if
they hit an error in one of the arguments, but instead produce an error
message and continue on to the next argument. (This is why error_msg()
sets toys.exitval to 1 if it's not already set, so that when we _do_
eventually exit it records the error.)

Should gzip do that? You have it exiting immediately... Huh, it looks
like the gnu/gnu/gnu/dammit version _does_ exit at the first error,
which seems awkward and wrong. So this matches the existing version, the
question remains what the behavior _should_ be.

And I'm over my half hour with half the file left to go. And I haven't
even spliced in the deflate plumbing from compress yet. :)

Rob
enh
2017-05-09 01:27:59 UTC
Permalink
Raw Message
Post by Rob Landley
Post by enh
if you're actually going to start to look, i'll attach my port of the
current toolbox implementation.
Ok, carving out a half hour to look at this... optargs should really
have [-123456789] at the end so "gzip -3 -9" is -9 but "gzip -9 -3" is
-3. While we're at it, OLDTOY() was there so these things could share
option strings, except why does zcat have -cd?
i just copied what the "real thing" does --- all three commands accept
the same set of options, and just ignore the ones they can't use. i
assumed _someone_ is relying on that. (i can't remember if i actually
hit such an instance --- i don't remember how it was that i realized
that they all accept all the options, but i do remember in the
beginning i had a different getopt string for each one, but changed
that even in the toolbox version.)
Post by Rob Landley
(And the three main
functions... if oldtoy isn't handling this right it needs fixing. Do you
care about the standalone builds, by the way? that was always the
complicating part, chording together shared infrastructure...)
i don't personally understand why someone would want one of these but
not all three, no.
Post by Rob Landley
I need to spend a 3 day weekend fixing the help infrastructure so it can
do reasonable includes, the duplication of -c and -f help text here
pains me.
the fact that the ps/top/pgrep help is just wrong pains me more :-)

note that -f isn't the same between all three, and the -c is slightly
different for all three. (and they'd differ more if we behaved more
like the "real thing" and documented accurately.)
Post by Rob Landley
Ok, jumping down to main()... it seems like there should be a
first_bit() function in lib/lib.c. Throw it in locally and move it when
I find a second user. (Leaving it _not_ static to remind me about the
library collation next time I hunt around for non-static functions.)
Setting FLAG_c for no arguments can't be right, assuming gzip is doing
what I think it is...
gunzip one.gz - three.gz < two.gz
And yes two gets output to stdout. So gunzip - writes "back" to stdout
automatically, so the loopfiles() behavior of "no arguments means one
argument equivalent to -" is fine and the callback has to handle that.
loopfiles() second argument is a function pointer, you don't need a
do_gz() wrapper you can just have the test in loopfiles to pass in the
right argument. (It's not changing partway through the loop...)
Hmmm, there's plumbing in things like patch and sed -i using
copy_tempfile() to create a temporary file we write to (and delete if
we're interrupted, via an atexit() variant). You're leaving the
half-finished output file if we get interrupted, is that what we should do?
Also, some commands go to a bit of trouble _not_ to exit immediately if
they hit an error in one of the arguments, but instead produce an error
message and continue on to the next argument. (This is why error_msg()
sets toys.exitval to 1 if it's not already set, so that when we _do_
eventually exit it records the error.)
Should gzip do that? You have it exiting immediately... Huh, it looks
like the gnu/gnu/gnu/dammit version _does_ exit at the first error,
which seems awkward and wrong. So this matches the existing version, the
question remains what the behavior _should_ be.
in the absence of a strong reason to do anything different, i just did
what i observed the "real thing" doing.

afaik the only differences are omissions.
Post by Rob Landley
And I'm over my half hour with half the file left to go. And I haven't
even spliced in the deflate plumbing from compress yet. :)
Rob
--
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
Rob Landley
2017-05-09 20:25:54 UTC
Permalink
Raw Message
Post by enh
Post by Rob Landley
Post by enh
if you're actually going to start to look, i'll attach my port of the
current toolbox implementation.
Ok, carving out a half hour to look at this... optargs should really
have [-123456789] at the end so "gzip -3 -9" is -9 but "gzip -9 -3" is
-3. While we're at it, OLDTOY() was there so these things could share
option strings, except why does zcat have -cd?
i just copied what the "real thing" does --- all three commands accept
the same set of options, and just ignore the ones they can't use. i
assumed _someone_ is relying on that. (i can't remember if i actually
hit such an instance --- i don't remember how it was that i realized
that they all accept all the options, but i do remember in the
beginning i had a different getopt string for each one, but changed
that even in the toolbox version.)
It's times like this where busybox is useful, since that's 10+ years of
an alternate implementation lots of people have had plenty of
opportunity to complain about. :)

(I vaguely recall that the IETF used to have a policy that standards
submissions had to have two different interoperable implementations.
This was a good policy.)

Checking busybox defconfig from february, zcat --help documents no
options...

./busybox zcat -c README.gz
./busybox zcat --fruitbasket README.gz

And is ignoring all options presented to it. (Blah. Proving nothing in
this instance.)

Hey, ubuntu's zcat will take non-gz filenames and look for a
corresponding gz file. (I.E. "zcat README" will display README.gz).
Should we do that? (Probably.) the man page says zcat is identical to
"gunzip -c", is this true here?

gzip README
gunzip -c README

Yup, that also finds the .gz version when not given a .gz. (The busybox
version doesn't have this trick though...)

Busybox gzip documents -dtcf but not the numbers. But it accepts the
numbers. And it's being specific about that:

$ ./busybox gzip -7 --potato README
gzip: unrecognized option '--potato'

Sigh. Ok, this time checking busybox was less useful than I'd hoped. :)
Post by enh
Post by Rob Landley
(And the three main
functions... if oldtoy isn't handling this right it needs fixing. Do you
care about the standalone builds, by the way? that was always the
complicating part, chording together shared infrastructure...)
i don't personally understand why someone would want one of these but
not all three, no.
Running "make test_zcat" does a standalone build of zcat, which should
act like zcat and not like gzip or gunzip. So there's one use case. :)
Post by enh
Post by Rob Landley
I need to spend a 3 day weekend fixing the help infrastructure so it can
do reasonable includes, the duplication of -c and -f help text here
pains me.
the fact that the ps/top/pgrep help is just wrong pains me more :-)
note that -f isn't the same between all three, and the -c is slightly
different for all three. (and they'd differ more if we behaved more
like the "real thing" and documented accurately.)
What does zcat -c _do_? (Make it _not_ act like zcat? As far as I can
tell ubuntu's is ignoring it.) How would -f behavior differ? (Is this in
the tests you sent?)

I see "acts slightly different" and hear "special case" and wonder "is
there any way we could not?" Which is where I have to read the code and
the man page and come up with test cases and wrap my head around _why_
and then see if there are users out there depending on this weirdness
(although mostly I check package configure and builds because I can find
and automate a lot of those) and...

I miss free time. It was nice.
Post by enh
Post by Rob Landley
Should gzip do that? You have it exiting immediately... Huh, it looks
like the gnu/gnu/gnu/dammit version _does_ exit at the first error,
which seems awkward and wrong. So this matches the existing version, the
question remains what the behavior _should_ be.
in the absence of a strong reason to do anything different, i just did
what i observed the "real thing" doing.
Oh sure. It's a great first pass, I'm just doing my normal gift horse
dentistry.

I try not to think in terms of "real thing", but instead the old IETF
model. The toybox implementation has the potential to become the "real
thing" for a lot of people in future. I'm trying to work out if there
_was_ a spec (I.E. if posix was functional), what it would look like?

Warn-but-continue is the common behavior elsewhere in toybox, because
that was common behavior in the existing utilities toybox was
implementing new versions of. I should survey the existing toybox
commands and see if there are _any_ that stop on error when handling
[FILE...] arguments, and if so (I don't remember any) compare the ubuntu
implementation to make sure it wasn't our mistake.

If gzip is the only oddball, I'd prefer to correct it in toybox. If
there are others, then it's not a special case...
Post by enh
afaik the only differences are omissions.
Yeah, but _should_ there be? Newbies learning the unixoid command line
are helped by consistency. (That's why in toybox everything accepts --
even "echo", regardless of what ubuntu did.) If future generations
learning this stuff _don't_ outnumber the current installed base, we're
doing it wrong.

And if we are going to clean up this sort of thing, the initial
introduction is the place to do it. If people need it, they'll complain.
(Later people would complain just because it changed.)

I can see you not wanting to field the bug reports, though. :)

Rob

P.S. Remember how I disabled --help output for "true" and "false"
because people complained? Bash has built-in "true" and "false"
implementations that behave like toybox does now, but ubuntu _also_ has
/bin/false and /bin/true behave like toybox _used_ to, and yes:

$ /bin/true --help > /dev/full
/bin/true: write error: No space left on device
$ echo $?
1

Moral of the story: the gnu/gnu/gnu/dammit stuff is not compatible with
_itself_. Second moral of the story:

$ /bin/true --help | wc -l
15
$ man true | wc -l
50

Yet that man page ends with:

SEE ALSO
The full documentation for true is maintained as a Texinfo
manual. If the info and true programs are properly installed
at your site, the command

info coreutils 'true invocation'

should give you access to the complete manual.

Because 15 lines of built-in help text plus 50 lines of man page is not
enough to fully describe the "true" command, for the _full_
documentation they need their bespoke proprietary documentation format
nobody else uses but they refuse to give up.

This sort of thing is why I don't see gnu versions of anything as "the
real thing", more what people used in the bad old days before we knew
better, uphill, both ways, in the snow.

Still Rob
enh
2017-05-09 22:08:04 UTC
Permalink
Raw Message
Post by Rob Landley
Post by enh
Post by Rob Landley
Post by enh
if you're actually going to start to look, i'll attach my port of the
current toolbox implementation.
Ok, carving out a half hour to look at this... optargs should really
have [-123456789] at the end so "gzip -3 -9" is -9 but "gzip -9 -3" is
-3. While we're at it, OLDTOY() was there so these things could share
option strings, except why does zcat have -cd?
i just copied what the "real thing" does --- all three commands accept
the same set of options, and just ignore the ones they can't use. i
assumed _someone_ is relying on that. (i can't remember if i actually
hit such an instance --- i don't remember how it was that i realized
that they all accept all the options, but i do remember in the
beginning i had a different getopt string for each one, but changed
that even in the toolbox version.)
It's times like this where busybox is useful, since that's 10+ years of
an alternate implementation lots of people have had plenty of
opportunity to complain about. :)
(I vaguely recall that the IETF used to have a policy that standards
submissions had to have two different interoperable implementations.
This was a good policy.)
Checking busybox defconfig from february, zcat --help documents no
options...
./busybox zcat -c README.gz
./busybox zcat --fruitbasket README.gz
And is ignoring all options presented to it. (Blah. Proving nothing in
this instance.)
well, i took that as proof that we probably should ignore all the
other arguments. (since both gnu and busybox did.)
Post by Rob Landley
Hey, ubuntu's zcat will take non-gz filenames and look for a
corresponding gz file. (I.E. "zcat README" will display README.gz).
Should we do that? (Probably.) the man page says zcat is identical to
"gunzip -c", is this true here?
gzip README
gunzip -c README
Yup, that also finds the .gz version when not given a .gz. (The busybox
version doesn't have this trick though...)
yeah, i noticed that but thought it was gross and used the fact that
busybox doesn't support it to say "YAGNI".

i should probably have added a note at the top of the file, but while
documenting deviations from POSIX makes a lot of sense, listing every
GNU wart isn't quite as obvious. (though i guess this would have been
a special and more interesting case: not just a GNU wart but one that
busybox doesn't share.)
Post by Rob Landley
Busybox gzip documents -dtcf but not the numbers. But it accepts the
$ ./busybox gzip -7 --potato README
gzip: unrecognized option '--potato'
Sigh. Ok, this time checking busybox was less useful than I'd hoped. :)
(which is maybe another argument for always documenting deliberate
differences: for the benefit of the next project to tread this path
:-) )
Post by Rob Landley
Post by enh
Post by Rob Landley
(And the three main
functions... if oldtoy isn't handling this right it needs fixing. Do you
care about the standalone builds, by the way? that was always the
complicating part, chording together shared infrastructure...)
i don't personally understand why someone would want one of these but
not all three, no.
Running "make test_zcat" does a standalone build of zcat, which should
act like zcat and not like gzip or gunzip. So there's one use case. :)
Post by enh
Post by Rob Landley
I need to spend a 3 day weekend fixing the help infrastructure so it can
do reasonable includes, the duplication of -c and -f help text here
pains me.
the fact that the ps/top/pgrep help is just wrong pains me more :-)
note that -f isn't the same between all three, and the -c is slightly
different for all three. (and they'd differ more if we behaved more
like the "real thing" and documented accurately.)
What does zcat -c _do_? (Make it _not_ act like zcat? As far as I can
tell ubuntu's is ignoring it.) How would -f behavior differ? (Is this in
the tests you sent?)
the thing i remember not implementing is that GNU at least checks
isatty. i don't remember whether busybox does. iirc i just have a TODO
in the tests saying "is there any way to write a test for this?".
Post by Rob Landley
I see "acts slightly different" and hear "special case" and wonder "is
there any way we could not?" Which is where I have to read the code and
the man page and come up with test cases and wrap my head around _why_
and then see if there are users out there depending on this weirdness
(although mostly I check package configure and builds because I can find
and automate a lot of those) and...
I miss free time. It was nice.
Post by enh
Post by Rob Landley
Should gzip do that? You have it exiting immediately... Huh, it looks
like the gnu/gnu/gnu/dammit version _does_ exit at the first error,
which seems awkward and wrong. So this matches the existing version, the
question remains what the behavior _should_ be.
in the absence of a strong reason to do anything different, i just did
what i observed the "real thing" doing.
Oh sure. It's a great first pass, I'm just doing my normal gift horse
dentistry.
I try not to think in terms of "real thing", but instead the old IETF
model. The toybox implementation has the potential to become the "real
thing" for a lot of people in future. I'm trying to work out if there
_was_ a spec (I.E. if posix was functional), what it would look like?
this was exactly why i _deleted_ the code for handling a missing
extension. i did implement it and wrote tests, but looked at what i'd
done and just felt like i wasn't doing the future any favors. (a quick
pop quiz showed that no one i work with knew about this wart.)
Post by Rob Landley
Warn-but-continue is the common behavior elsewhere in toybox, because
that was common behavior in the existing utilities toybox was
implementing new versions of. I should survey the existing toybox
commands and see if there are _any_ that stop on error when handling
[FILE...] arguments, and if so (I don't remember any) compare the ubuntu
implementation to make sure it wasn't our mistake.
If gzip is the only oddball, I'd prefer to correct it in toybox. If
there are others, then it's not a special case...
Post by enh
afaik the only differences are omissions.
Yeah, but _should_ there be? Newbies learning the unixoid command line
are helped by consistency. (That's why in toybox everything accepts --
even "echo", regardless of what ubuntu did.) If future generations
learning this stuff _don't_ outnumber the current installed base, we're
doing it wrong.
And if we are going to clean up this sort of thing, the initial
introduction is the place to do it. If people need it, they'll complain.
(Later people would complain just because it changed.)
I can see you not wanting to field the bug reports, though. :)
"why is so much stuff missing from top/ps/pgrep?" is my mostly
frequent bug report atm, because people trust that --help is telling
them the whole truth :-)
Post by Rob Landley
Rob
P.S. Remember how I disabled --help output for "true" and "false"
because people complained? Bash has built-in "true" and "false"
implementations that behave like toybox does now, but ubuntu _also_ has
$ /bin/true --help > /dev/full
/bin/true: write error: No space left on device
$ echo $?
1
Moral of the story: the gnu/gnu/gnu/dammit stuff is not compatible with
$ /bin/true --help | wc -l
15
$ man true | wc -l
50
SEE ALSO
The full documentation for true is maintained as a Texinfo
manual. If the info and true programs are properly installed
at your site, the command
info coreutils 'true invocation'
should give you access to the complete manual.
Because 15 lines of built-in help text plus 50 lines of man page is not
enough to fully describe the "true" command, for the _full_
documentation they need their bespoke proprietary documentation format
nobody else uses but they refuse to give up.
This sort of thing is why I don't see gnu versions of anything as "the
real thing", more what people used in the bad old days before we knew
better, uphill, both ways, in the snow.
the reason i tend to say "real thing" rather than "gnu" is just
because i'm too lazy to check. i happen to know bzip2 isn't gnu, for
example, and i'm pretty sure (because i maintain Android's zlib) that
gzip is gnu (because it's not part of the zlib distribution, though
they do have a cut-down version), and unzip/zip probably aren't gnu
(even though they're not in the zlib distribution either) because they
just don't "feel" like gnu tools.

i've also been known to say "desktop" or "traditional", but both of
those alternatives have their problems too.
Post by Rob Landley
Still Rob
--
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
enh
2017-05-09 16:02:09 UTC
Permalink
Raw Message
Post by Rob Landley
Post by enh
Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
another day,
Blah, I should take this one.
I have deflate decompression side implemented already (and working at
one point, although I just tested "make zcat" and got a bad crc
decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
compression side implemented and somewhere around here is a version
that's about 2/3 through. I stopped at "where do dictionary resets
happen" and basically go the answer "nobody seems to agree, just do it
every 250k". (Which means my gzip and other gzip won't necessarily
produce the same binary, but it should extract the same. There's
actually a format hitch you can do sticking in gratuitous resets to
allow parallel decompression, it's a bit like mp4 streaming mode
sticking in extra keyframes, only noticeably cheaper. :)
Adding zlib mode to that is basically the same plumbing as
toys/*/md5sum.c does for libcrypto.
Lemme try to find a chunk of time to devote to that...
Post by enh
but at least the tests are easy...
(These tests pass with TEST_HOST and on the toolbox versions, but the
toybox toys are in pending and very broken.)
The problem is this is just testing that gzip and gunzip reverse each
other. If they share infrastructure and screw something up the same way,
what have you proved? Also you can't test gunzip without gzip being present.
yeah, despite being the person who's trying to run the toybox tests on
a toybox-only system, i relied on testing on the host.

commit these as better than nothing and i'll follow up with a patch
adding pre-canned inputs for each of them, so each test only needs one
tool?

when i do create the pre-canned inputs, do you have any preference
about which tool (toybox/busybox/gnu) generates them?
Post by Rob Landley
For bzip I added a tests/files/bzcat (and already had a lot of
tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
files for reference under tests/files.
I'll try to take a closer look at this later...
Rob
--
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
enh
2017-05-22 17:50:14 UTC
Permalink
Raw Message
Post by enh
Post by Rob Landley
Post by enh
Bringing the zlib-based gzip/gunzip/zcat over to toybox is a problem for
another day,
Blah, I should take this one.
I have deflate decompression side implemented already (and working at
one point, although I just tested "make zcat" and got a bad crc
decompressing the musl-1.1.16.gz tarball). I have the first 1/3 or so of
compression side implemented and somewhere around here is a version
that's about 2/3 through. I stopped at "where do dictionary resets
happen" and basically go the answer "nobody seems to agree, just do it
every 250k". (Which means my gzip and other gzip won't necessarily
produce the same binary, but it should extract the same. There's
actually a format hitch you can do sticking in gratuitous resets to
allow parallel decompression, it's a bit like mp4 streaming mode
sticking in extra keyframes, only noticeably cheaper. :)
Adding zlib mode to that is basically the same plumbing as
toys/*/md5sum.c does for libcrypto.
Lemme try to find a chunk of time to devote to that...
Post by enh
but at least the tests are easy...
(These tests pass with TEST_HOST and on the toolbox versions, but the
toybox toys are in pending and very broken.)
The problem is this is just testing that gzip and gunzip reverse each
other. If they share infrastructure and screw something up the same way,
what have you proved? Also you can't test gunzip without gzip being present.
yeah, despite being the person who's trying to run the toybox tests on
a toybox-only system, i relied on testing on the host.
commit these as better than nothing and i'll follow up with a patch
adding pre-canned inputs for each of them, so each test only needs one
tool?
ping? (and next question too for the follow-on patch.)
Post by enh
when i do create the pre-canned inputs, do you have any preference
about which tool (toybox/busybox/gnu) generates them?
Post by Rob Landley
For bzip I added a tests/files/bzcat (and already had a lot of
tests/files/blkid/*.bz2). I'd like to have at least a couple canned .gz
files for reference under tests/files.
I'll try to take a closer look at this later...
Rob
--
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
--
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.
Loading...