[Toybox] Numeric values in dd operands

On 02/20/2018 08:32 AM, toybox-***@lists.landley.net wrote:
/cut

1. Re: Numeric values in dd operands (Rob Landley)

/cut

----------------------------------------------------------------------
Message: 1
Date: Mon, 19 Feb 2018 11:30:11 -0600
Subject: Re: [Toybox] Numeric values in dd operands
Content-Type: text/plain; charset=windows-1252
And permission to reply to this publicly recieved, so...

Sigh, so many people using code out of pending. My local changes to this file
$ git diff toys/*/dd.c | diffstat
dd.c | 96 ++++++++++++++++++++++++++++++++++++++++---------------------------
1 file changed, 58 insertions(+), 38 deletions(-)
and that's just "the next pass"...

We have scripts that assume it is OK to pass a decimal number with leading zeros
to dd, so we have a problem.

Indeed. I had common code to handle the suffixes but it also does the c-standard
base detection. Which is a side effect you don't want. Hmmm...

The dd on ubuntu and as specified in the posix spec
(http://pubs.opengroup.org/onlinepubs/9699919799/) says the numbers are decimal

The posix spec says that the command supports ebcdic to ascii conversions. I've
been taking a somewhat loose interpretation of this one. :)

and allows various suffixes, e.g. k or b, and a mid-number multiplier ?x? ?so
12x3 is decimal 36, 017 is 17 (not 15) , ?0x999 is 0 and 1kx1k is 1048576

Are you actually using that mid-number multiplier? I was asking on the list last
year if anyone anywhere actually did that. (It's a relic from before the shell
provided $((123*456)).)

I always interpreted it as the ability for someone putting an double
character, such as kb in instead of just a k, I have seen 12gb316 used
(meaning 12,316,000,000) in the auto-output for raid drive stats before
today. I hadn't thought of it being a multiplier character.

The 0.7.5 implementation assumes that x is part of a hexadecimal prefix so 0x12
is interpreted as 18 rather than 0, and 3x12 is an error.

And 3x12 could be interpreted as 3 to the power 12 of whatever base is
being used as it would be in calculus. However a leading 0 (0x12) would
only define, not say, it would depend on the base being used, not just
hexidecimal. It could also be quaternary (base4), octal (base8), or even
radix (64bit), all used in computation and hardware data collection
input circuits. I wonder how you would define/control this?

I'm reluctant to add an argument to a library function that's currently used 57
times without it. If the problem is just leading zeroes, a trivial wrapper to
"while (*str=='0') str++;" in _this_ command should fix it?
I agree that octal is obsolete (outside of file permissions), and if posix says
it's decimal then leading zeroes changing that is surprising and should probably
be fixed. I'm less convinced about the x thing (which means not supporting
hexadecimal, which is _not_ obsolete...)

$ ping 008.008.008.008
ping: unknown host 008.008.008.008
$ ping 007.007.007.007
PING 007.007.007.007 (7.7.7.7) 56(84) bytes of data.
^C
$ ping 010.010.010.010
PING 010.010.010.010 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=220 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=48.4 ms
That's a leading 0 indicating octal in an ipv4 command line utility...
$ dpkg-query -S $(which ping)
iputils-ping: /bin/ping
Rob

/cut

sorry rob, but...

scsijon

Rob Landley

2018-02-20 17:28:38 UTC

Post by Rob Landley
Are you actually using that mid-number multiplier? I was asking on the list last
year if anyone anywhere actually did that. (It's a relic from before the shell
provided $((123*456)).)

I always interpreted it as the ability for someone putting an double character,
such as kb in instead of just a k, I have seen 12gb316 used (meaning
12,316,000,000) in the auto-output for raid drive stats before today. I hadn't
thought of it being a multiplier character.

Huh. That's an interesting non-posix extension I hadn't heard of before.

The 0.7.5 implementation assumes that x is part of a hexadecimal prefix so 0x12
is interpreted as 18 rather than 0, and 3x12 is an error.

And 3x12 could be interpreted as 3 to the power 12 of whatever base is being
used as it would be in calculus.

Not according to http://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html

For the bs=, cbs=, ibs=, and obs= operands, the application shall supply an
A positive decimal number
A positive decimal number followed by k, specifying multiplication by 1024
A positive decimal number followed by b, specifying multiplication by 512
Two or more positive decimal numbers (with or without k or b) separated by
x, specifying the product of the indicated values
All of the operands are processed before any input is read.

Product means multiply.

I'm not _against_ extensions, toybox is also handling megabyte and gigabyte and
such because it's not the 1980's anymore. (And blocks went from 512 bytes to 4k
ten years ago: https://lwn.net/Articles/322777/ but that's _far_ too recent for
posix to even be aware it's happened, let alone respond to it.)

However a leading 0 (0x12) would only define,
not say, it would depend on the base being used, not just hexidecimal. It could
also be quaternary (base4), octal (base8), or even radix (64bit),

I didn't make the 0 means octal and 0x means hexadecimal prefixes up:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/strtol.html

It's already widely implemented in the Linux command line. Even though posix
doesn't say "printf %d 0x1234" should print 4660, ubuntu's printf does (and yes
that also means 01234 prints as 668).

Every toybox command that takes a number argument has been doing this for years.
You can go "head -n 0xa". This is the first complaint about it so far.

In toybox I try to err on the side of _consistency_. All the commands behave the
_same_ way. This would be an explicit exception where dd count=01234 is _not_
interpreted as 668, meaning dd is a special case different from everything else.

all used in computation and hardware data collection input circuits.

But not in c99 or posix.

I wonder how you would define/control this?

I wouldn't?

Base 4 output has never historically been part of any unix command I'm aware of.
It's not in c99, it's not in posix, it's not in the linux standard base, it's
not in busybox, it's not in ubuntu's current command line, it's not in my old
the Red hat 9 qemu image (which I flung on https://busybox.net/downloads/qemu/
years ago and is strangely enough still there, although the README isn't. user
busybox password busybox, I think that's the root password too.)

I'm not trying to make up new stuff, I'm trying to serve an existing userbase
base of people who are part of a 50 year tradition. (The first pdp-7 unix was
written in 1969, the 50th anniversary is next year.) The best way to serve them
is to be consistent with historical practice.

I'm also trying to provide something new users can learn easily. The best way to
serve _them_ is provide something consistent so they only have to learn a trick
once and then it works the same way everywhere.

When consistency and historical practice collide, the answer is not always
obvious to me.

I've often chosen to implement only some of the posix spec, because portions of
posix are deeply obsolete. It specifies the "sccs" source control system, batch
control commands (qdel and friends), fortran 77, the "ed" line editor, uucp. The
roadmap.html page has a section devoted to this.

Sometimes you have to explicitly break posix: it says zcat undoes "compress"
format (adaptive Lempel-Ziv coding, which was patented in 1984 and utterly dead
by the time that expired). Everybody else has zcat undo "deflate", the algorithm
introduced by pkzip 2.x in 1993. I care about current reality, not what posix
says to do. I only care about posix when it documents current reality.

In dd posix specifies ebcdic to ascii conversions, the "swap" byte swapping
option (assuming only 16 bit systems have endianness issues), ucase/lcase case
mapping that does not _conceptually_ work with multibyte encodings like (let's
chop that data into blocks and then do case conversion across block boundaries
without ever looking at data from another block...)

A real user piped up and said their existing script doesn't work with my tool.
That feedback is of interest to me.

Rob

enh

2018-02-20 17:32:30 UTC

I always interpreted it as the ability for someone putting an double character,
such as kb in instead of just a k, I have seen 12gb316 used (meaning
12,316,000,000) in the auto-output for raid drive stats before today. I hadn't
thought of it being a multiplier character.

Huh. That's an interesting non-posix extension I hadn't heard of before.

The 0.7.5 implementation assumes that x is part of a hexadecimal prefix so 0x12
is interpreted as 18 rather than 0, and 3x12 is an error.

And 3x12 could be interpreted as 3 to the power 12 of whatever base is being
used as it would be in calculus.

Not according to http://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html

Product means multiply.
I'm not _against_ extensions, toybox is also handling megabyte and gigabyte and
such because it's not the 1980's anymore. (And blocks went from 512 bytes to 4k
ten years ago: https://lwn.net/Articles/322777/ but that's _far_ too recent for
posix to even be aware it's happened, let alone respond to it.)

However a leading 0 (0x12) would only define,
not say, it would depend on the base being used, not just hexidecimal. It could
also be quaternary (base4), octal (base8), or even radix (64bit),

http://pubs.opengroup.org/onlinepubs/9699919799/functions/strtol.html
It's already widely implemented in the Linux command line. Even though posix
doesn't say "printf %d 0x1234" should print 4660, ubuntu's printf does (and yes
that also means 01234 prints as 668).
Every toybox command that takes a number argument has been doing this for years.
You can go "head -n 0xa". This is the first complaint about it so far.
In toybox I try to err on the side of _consistency_. All the commands behave the
_same_ way. This would be an explicit exception where dd count=01234 is _not_
interpreted as 668, meaning dd is a special case different from everything else.

all used in computation and hardware data collection input circuits.

But not in c99 or posix.

I wonder how you would define/control this?

I wouldn't?
Base 4 output has never historically been part of any unix command I'm aware of.
It's not in c99, it's not in posix, it's not in the linux standard base, it's
not in busybox, it's not in ubuntu's current command line, it's not in my old
the Red hat 9 qemu image (which I flung on https://busybox.net/downloads/qemu/
years ago and is strangely enough still there, although the README isn't. user
busybox password busybox, I think that's the root password too.)
I'm not trying to make up new stuff, I'm trying to serve an existing userbase
base of people who are part of a 50 year tradition. (The first pdp-7 unix was
written in 1969, the 50th anniversary is next year.) The best way to serve them
is to be consistent with historical practice.
I'm also trying to provide something new users can learn easily. The best way to
serve _them_ is provide something consistent so they only have to learn a trick
once and then it works the same way everywhere.
When consistency and historical practice collide, the answer is not always
obvious to me.
I've often chosen to implement only some of the posix spec, because portions of
posix are deeply obsolete. It specifies the "sccs" source control system, batch
control commands (qdel and friends), fortran 77, the "ed" line editor, uucp. The
roadmap.html page has a section devoted to this.
Sometimes you have to explicitly break posix: it says zcat undoes "compress"
format (adaptive Lempel-Ziv coding, which was patented in 1984 and utterly dead
by the time that expired). Everybody else has zcat undo "deflate", the algorithm
introduced by pkzip 2.x in 1993. I care about current reality, not what posix
says to do. I only care about posix when it documents current reality.
In dd posix specifies ebcdic to ascii conversions, the "swap" byte swapping
option (assuming only 16 bit systems have endianness issues), ucase/lcase case
mapping that does not _conceptually_ work with multibyte encodings like (let's
chop that data into blocks and then do case conversion across block boundaries
without ever looking at data from another block...)
A real user piped up and said their existing script doesn't work with my tool.
That feedback is of interest to me.

and as the person who set us down the strtol path
(https://github.com/landley/toybox/commit/d5088a059649daf34e729995bb3daa3eb64fa432#diff-ce001a87e82f850a38fd93183e12b417),
the original request i had was just for hex. like you say, no-one's
used octal (on purpose) for anything other than mode for decades now.

Post by Rob Landley
Rob
_______________________________________________
Toybox mailing list
http://lists.landley.net/listinfo.cgi/toybox-landley.net

Rob Landley

2018-02-20 17:57:28 UTC

Post by Rob Landley
A real user piped up and said their existing script doesn't work with my tool.
That feedback is of interest to me.

I'm tempted to have atolx() skip leading zeroes, and then base 16 if the first
character is an x and base 10 otherwise. Except supporting the - basically means
open coding the sucker...

Anyway, my _real_ question is, if I'm yanking octal from dd= should I yank it
from everywhere? I still think it's useful for "printf %d 0123" to be able to
cope with it (especially since "printf %o 668" prints 1234). And maybe $((0123))
needs it too?

No clue where to draw the line on this one. Hmmm...

Rob

enh

2018-02-20 18:20:41 UTC

Post by Rob Landley
A real user piped up and said their existing script doesn't work with my tool.
That feedback is of interest to me.

I'm tempted to have atolx() skip leading zeroes, and then base 16 if the first
character is an x and base 10 otherwise. Except supporting the - basically means
open coding the sucker...
Anyway, my _real_ question is, if I'm yanking octal from dd= should I yank it
from everywhere? I still think it's useful for "printf %d 0123" to be able to
cope with it (especially since "printf %o 668" prints 1234). And maybe $((0123))
needs it too?
No clue where to draw the line on this one. Hmmm...

you could make it opt-in and add octal support to places where its
absence is actually noticed.

Post by Rob Landley
Rob

Robert Thompson

2018-02-20 20:56:35 UTC

I've seen a fair number of scripts that use printf's ability to
interpret hex and octal arguments. This is often combined with
printf's ability to format outputs into decimal, octal, or hex, to
make a convenient base-shifter. For this discussion, it's the argument
conversion, not the formatting that's relevant...

Working from memory, I've seen patterns such as those below, often in
$() (or ` `) expansions. In some cases, the converted number was
roundtripped during the lifetime of the script.

printf "%u\n" 0377
255
(I've seen this one using both the "unsigned integer" and the various
printf(3) integer formatters)

printf "%o\n" 0x1b
33

printf "%#o\n" 0x1b
033

printf "%x\n" 77
4d

printf "0x%x\n" 77
0x4d

printf "%#x\n" 77
0x4d

Post by Rob Landley
A real user piped up and said their existing script doesn't work with my tool.
That feedback is of interest to me.

I'm tempted to have atolx() skip leading zeroes, and then base 16 if the first
character is an x and base 10 otherwise. Except supporting the - basically means
open coding the sucker...
Anyway, my _real_ question is, if I'm yanking octal from dd= should I yank it
from everywhere? I still think it's useful for "printf %d 0123" to be able to
cope with it (especially since "printf %o 668" prints 1234). And maybe $((0123))
needs it too?
No clue where to draw the line on this one. Hmmm...

you could make it opt-in and add octal support to places where its
absence is actually noticed.

Post by Rob Landley
Rob

_______________________________________________
Toybox mailing list
http://lists.landley.net/listinfo.cgi/toybox-landley.net

scsijon

2018-02-20 23:08:22 UTC

fyi,

Both quaternary (base4) and octal (base8) are used by analog device
inputs even today. There's a lot of security systems out there! And you
can even get them storing and using the data stacked in the rom, where
octal can appear as hexidecimal but are really upper and lower octal
pairs, such as resistance tolerance ranges for tape and switches. I
admit that after some research it seeems most of these devices are now
octal so maybe I should not have brought quaternary up.

I was asked abour radix (64bit), as it seems too be hard to find data
on. It's written in double octal and used at least internally (register
to register) in a number of raid systems and if you know what your doing
you can use these registers to speed throughput up or change data
priority. It's also used in Mainframe timeslot control and there is the
code for linux timeslots available for a number of mainframes but you
do(did) need implicit approval from LinusT to implement it. All part of
somethings I did in my previous work-life.

And i'll leave it at that having most likely confused a lot of the
group. My apologies.

scsijon

I always interpreted it as the ability for someone putting an double character,
such as kb in instead of just a k, I have seen 12gb316 used (meaning
12,316,000,000) in the auto-output for raid drive stats before today. I hadn't
thought of it being a multiplier character.

Huh. That's an interesting non-posix extension I hadn't heard of before.