Discussion:
getconf NPROCESSORS_ONLN broken
(too old to reply)
enh
2018-09-28 21:41:35 UTC
Permalink
$ ./toybox getconf NPROCESSORS_CONF NPROCESSORS_ONLN
48
2097152

seems like this isn't working:

# Extract names, remove blank lines, filter, replace unknown #defines
# with UNKNOWN
sed -n "/char [*]${1}_names[[]/"',/^}/s/[^"]*"\([^"]*\) *",*/\1\n/pg' \
toys/posix/getconf.c | grep -v '^$' | $2 |
sed -e "$DEFINES" -e "t;d;a UNKNOWN" | xargs | tr ' ' ',' &&
echo '};'

glibc doesn't have an _XOPEN_UUCP but instead of the promised UNKNOWN,
everything after that is just off-by-one...
enh
2018-09-29 03:05:11 UTC
Permalink
As someone who has a vested interest in having less generated code (because
I'd like to stop checking in generated files and use toybox in the AOSP
build[1])... do we even need to do this? How many of these defines are
missing on glibc/musl/bionic? If it's only one or two as I assume, could we
just have the #ifdefs instead?

Or are there actually a large number of these missing?

___
1. The real problem for me there is that I don't want to replicate all your
scripts in the AOSP build system, but I do want to do the equivalent work.
If it were just the help and flags, I'd probably swallow it --- the library
availability is something I can hard code anyway. But random stuff like
this is something I'd much rather avoid...
Post by enh
$ ./toybox getconf NPROCESSORS_CONF NPROCESSORS_ONLN
48
2097152
# Extract names, remove blank lines, filter, replace unknown #defines
# with UNKNOWN
sed -n "/char [*]${1}_names[[]/"',/^}/s/[^"]*"\([^"]*\) *",*/\1\n/pg' \
toys/posix/getconf.c | grep -v '^$' | $2 |
sed -e "$DEFINES" -e "t;d;a UNKNOWN" | xargs | tr ' ' ',' &&
echo '};'
glibc doesn't have an _XOPEN_UUCP but instead of the promised UNKNOWN,
everything after that is just off-by-one...
enh
2018-10-02 21:11:13 UTC
Permalink
(in the hope that silence means you don't hate the idea, i'll send a patch
implementing this...)
Post by enh
As someone who has a vested interest in having less generated code
(because I'd like to stop checking in generated files and use toybox in the
AOSP build[1])... do we even need to do this? How many of these defines are
missing on glibc/musl/bionic? If it's only one or two as I assume, could we
just have the #ifdefs instead?
Or are there actually a large number of these missing?
___
1. The real problem for me there is that I don't want to replicate all
your scripts in the AOSP build system, but I do want to do the equivalent
work. If it were just the help and flags, I'd probably swallow it --- the
library availability is something I can hard code anyway. But random stuff
like this is something I'd much rather avoid...
Post by enh
$ ./toybox getconf NPROCESSORS_CONF NPROCESSORS_ONLN
48
2097152
# Extract names, remove blank lines, filter, replace unknown #defines
# with UNKNOWN
sed -n "/char [*]${1}_names[[]/"',/^}/s/[^"]*"\([^"]*\) *",*/\1\n/pg' \
toys/posix/getconf.c | grep -v '^$' | $2 |
sed -e "$DEFINES" -e "t;d;a UNKNOWN" | xargs | tr ' ' ',' &&
echo '};'
glibc doesn't have an _XOPEN_UUCP but instead of the promised UNKNOWN,
everything after that is just off-by-one...
Rob Landley
2018-10-04 20:26:08 UTC
Permalink
Post by enh
(in the hope that silence means you don't hate the idea, i'll send a patch
implementing this...)
The silence meant my netbook died, but I don't hate the idea. I'd really rather
not reintroduce the #ifdef staircase where each symbol name appears multiple
times, but I admit my "simple hack to avoid that" turned out to be a lot less
simple than I thought it would. :P

Rob
enh
2018-10-04 20:48:09 UTC
Permalink
yeah, and right now we have exactly one macro for which we need this...

...and an alternative fix would be to just drop the UUCP feature from
getconf. GNU getconf doesn't support it (somewhat obviously), and
Android is always going to say "no", and i'm not expecting a great
UUCP renaissance any time soon where it suddenly becomes relevant
again.

(but i'm still interested to see that's even possible with the
preprocessor. i've long believed it wasn't.)
Post by Rob Landley
Post by enh
(in the hope that silence means you don't hate the idea, i'll send a patch
implementing this...)
The silence meant my netbook died, but I don't hate the idea. I'd really rather
not reintroduce the #ifdef staircase where each symbol name appears multiple
times, but I admit my "simple hack to avoid that" turned out to be a lot less
simple than I thought it would. :P
Rob
Rob Landley
2018-10-04 21:12:16 UTC
Permalink
Post by enh
yeah, and right now we have exactly one macro for which we need this...
...and an alternative fix would be to just drop the UUCP feature from
getconf. GNU getconf doesn't support it (somewhat obviously), and
Android is always going to say "no", and i'm not expecting a great
UUCP renaissance any time soon where it suddenly becomes relevant
again.
Trimming the UUCP feature sounds fine to me, although I need to revisit the
whole mess of:

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/getconf.html
http://pubs.opengroup.org/onlinepubs/9699919799/functions/sysconf.html
http://pubs.opengroup.org/onlinepubs/9699919799/functions/confstr.html

At some point...

Possibly what I need is an #ifdef/#else staircase for all those symbols in a
header file, which can be generated once from the actual posix spec and then
ignored. (I don't care about ugly in header files. I just don't want the data in
two places that have to be kept in sync...)
Post by enh
(but i'm still interested to see that's even possible with the
preprocessor. i've long believed it wasn't.)
Preprocessor combined with the ? : operator working on a constant first argument
so the test optimizes out, maybe? (If C11 or C18 had added _is_defined(x) and
macro_or_default(x, y) instead of all the useless nonsense it _did_ add, I'd
care a lot more about them...)

But at the moment I'm tired after a long day of sitting in a cubicle and trying
to work through company politics and history to figure out what (if anything)
the technical requirements for various Change Requests are, besides which I
shouldn't be thinking about this for another hour anyway. :)

Rob
enh
2018-10-04 23:06:14 UTC
Permalink
Post by Rob Landley
Post by enh
yeah, and right now we have exactly one macro for which we need this...
...and an alternative fix would be to just drop the UUCP feature from
getconf. GNU getconf doesn't support it (somewhat obviously), and
Android is always going to say "no", and i'm not expecting a great
UUCP renaissance any time soon where it suddenly becomes relevant
again.
Trimming the UUCP feature sounds fine to me, although I need to revisit the
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/getconf.html
http://pubs.opengroup.org/onlinepubs/9699919799/functions/sysconf.html
http://pubs.opengroup.org/onlinepubs/9699919799/functions/confstr.html
At some point...
Possibly what I need is an #ifdef/#else staircase for all those symbols in a
header file, which can be generated once from the actual posix spec and then
ignored. (I don't care about ugly in header files. I just don't want the data in
two places that have to be kept in sync...)
i think that's overkill. we demonstrably don't have to care about
anything outside the subset that GNU supports. (and in practice it
seems like there are really only a tiny handful that are actually
used.)

i think my patch to add the pathconf variables is more useful ---
there are real live instances of that in the AOSP build.

note that i also have an orthogonal interest in going in the direction
of my patches: if i'm going to move AOSP to toybox, i'll end up
replicating your build scripts in soong. so as far as i'm concerned,
the less that's done there, the better...
Post by Rob Landley
Post by enh
(but i'm still interested to see that's even possible with the
preprocessor. i've long believed it wasn't.)
Preprocessor combined with the ? : operator working on a constant first argument
so the test optimizes out, maybe? (If C11 or C18 had added _is_defined(x) and
macro_or_default(x, y) instead of all the useless nonsense it _did_ add, I'd
care a lot more about them...)
But at the moment I'm tired after a long day of sitting in a cubicle and trying
to work through company politics and history to figure out what (if anything)
the technical requirements for various Change Requests are, besides which I
shouldn't be thinking about this for another hour anyway. :)
Rob
Rob Landley
2018-10-05 03:18:01 UTC
Permalink
Post by enh
Post by Rob Landley
Possibly what I need is an #ifdef/#else staircase for all those symbols in a
header file, which can be generated once from the actual posix spec and then
ignored. (I don't care about ugly in header files. I just don't want the data in
two places that have to be kept in sync...)
i think that's overkill. we demonstrably don't have to care about
anything outside the subset that GNU supports. (and in practice it
seems like there are really only a tiny handful that are actually
used.)
Indeed, but it's hard to predict what that handful _is_. It's one of them
"nobody uses more than 20% of the features, but they're never the same 20%"
things...

An if/else staircase in the header _and_ a list of symbols in the command is
exactly what I was trying to avoid. That said, an if/else staircase in the
command is at least contained in one place.

Or would could just say "this is tested against bionic, musl, and glibc, and
anything else can define the symbols or add itself to portability.h". I have a
vague desire to make toybox build against older ndk apis, but am unlikely to
have bandwidth before they're the rest of the way obsolete. (Modulo I'm not sure
kit-kat will ever die at this point.)
Post by enh
i think my patch to add the pathconf variables is more useful ---
there are real live instances of that in the AOSP build.
I'll probably apply it as-is tomorrow, it's just one of those diffs that removes
enough code it's hard to read through, and I'm out of brain to think through it
today. (Didn't get home until 6 and had a 7:30 appointment just now wrapping up...)
Post by enh
note that i also have an orthogonal interest in going in the direction
of my patches: if i'm going to move AOSP to toybox, i'll end up
replicating your build scripts in soong. so as far as i'm concerned,
the less that's done there, the better...
The makefile tries hard to just be a wrapper, it boils down to a call to
scripts/make.sh. You can "make clean; make defconfig; scripts/make.sh"
And "make sed" and "make test_sed" stuff is calls to "scripts/single.sh sed" and
"scripts/test.sh sed". I.E. the build is written almost entirely in bash. :)

I should split scripts/ up into the actual build plumbing, a directory of tools
like bloatcheck and findglobals and showasm, and the test suite plumbing.

Hmmm, possibly the test suite plumbing should move under tests/ somehow...

In terms of eliminating unnecessary build plumbing, the tagged array stuff
(scripts/mktags.c) is the other big chunk of preprocessing, which is really
there to make a combined array/enum construct where you have a string name for
each entry _and_ a symbol of the same name resolving to the index of that entry
in the array. I would LOVE a way to do this with macros instead. (I can
stringify, but how do I get an increasing symbol number that goes up by one each
time it's referenced? I'd need INCREMENT(previous) to nest +1+1+1+1 and it's not
obvious how, which is prsumably why gcc has a __COUNTER__ macro, which is fiddly
and which I dunno if llvm implements it...

https://gcc.gnu.org/onlinedocs/cpp/Common-Predefined-Macros.html

Ah yes, it does:

https://clang.llvm.org/docs/LanguageExtensions.html#builtin-macros

I've been doing things via preprocessing on the assumption they'd be portable.
(It's either C code that runs on the host, or it's using standard command line
utilities that toybox itself implements...) But if things like __COUNTER__ are
better for you I'll see if I can simplify?

(I can also break up make.sh into multiple smaller scripts if that would help...?)

Rob
enh
2018-10-05 19:54:31 UTC
Permalink
Post by Rob Landley
Post by enh
Post by Rob Landley
Possibly what I need is an #ifdef/#else staircase for all those symbols in a
header file, which can be generated once from the actual posix spec and then
ignored. (I don't care about ugly in header files. I just don't want the data in
two places that have to be kept in sync...)
i think that's overkill. we demonstrably don't have to care about
anything outside the subset that GNU supports. (and in practice it
seems like there are really only a tiny handful that are actually
used.)
Indeed, but it's hard to predict what that handful _is_. It's one of them
"nobody uses more than 20% of the features, but they're never the same 20%"
things...
don't worry, i was planning on sending you missing ones. i just don't
want my patches to get too far out of sync with upstream.
Post by Rob Landley
An if/else staircase in the header _and_ a list of symbols in the command is
exactly what I was trying to avoid. That said, an if/else staircase in the
command is at least contained in one place.
Or would could just say "this is tested against bionic, musl, and glibc, and
anything else can define the symbols or add itself to portability.h". I have a
vague desire to make toybox build against older ndk apis, but am unlikely to
have bandwidth before they're the rest of the way obsolete. (Modulo I'm not sure
kit-kat will ever die at this point.)
not relevant --- since NDK r14 ("unified headers") you get the same
headers regardless of API level. that already "just works".
Post by Rob Landley
Post by enh
i think my patch to add the pathconf variables is more useful ---
there are real live instances of that in the AOSP build.
I'll probably apply it as-is tomorrow, it's just one of those diffs that removes
enough code it's hard to read through, and I'm out of brain to think through it
today. (Didn't get home until 6 and had a 7:30 appointment just now wrapping up...)
Post by enh
note that i also have an orthogonal interest in going in the direction
of my patches: if i'm going to move AOSP to toybox, i'll end up
replicating your build scripts in soong. so as far as i'm concerned,
the less that's done there, the better...
The makefile tries hard to just be a wrapper, it boils down to a call to
scripts/make.sh. You can "make clean; make defconfig; scripts/make.sh"
And "make sed" and "make test_sed" stuff is calls to "scripts/single.sh sed" and
"scripts/test.sh sed". I.E. the build is written almost entirely in bash. :)
I should split scripts/ up into the actual build plumbing, a directory of tools
like bloatcheck and findglobals and showasm, and the test suite plumbing.
Hmmm, possibly the test suite plumbing should move under tests/ somehow...
In terms of eliminating unnecessary build plumbing, the tagged array stuff
(scripts/mktags.c) is the other big chunk of preprocessing, which is really
there to make a combined array/enum construct where you have a string name for
each entry _and_ a symbol of the same name resolving to the index of that entry
in the array. I would LOVE a way to do this with macros instead. (I can
stringify, but how do I get an increasing symbol number that goes up by one each
time it's referenced? I'd need INCREMENT(previous) to nest +1+1+1+1 and it's not
obvious how, which is prsumably why gcc has a __COUNTER__ macro, which is fiddly
and which I dunno if llvm implements it...
i'm kind of resigned to having to explicitly build the help, flags,
and arrays binaries and run them, and they all seem like they're
pulling their weight anyway. i just don't want to encourage too much
random other stuff, because i need to replicate all your shell scripts
in soong.
Post by Rob Landley
https://gcc.gnu.org/onlinedocs/cpp/Common-Predefined-Macros.html
https://clang.llvm.org/docs/LanguageExtensions.html#builtin-macros
I've been doing things via preprocessing on the assumption they'd be portable.
(It's either C code that runs on the host, or it's using standard command line
utilities that toybox itself implements...) But if things like __COUNTER__ are
better for you I'll see if I can simplify?
(I can also break up make.sh into multiple smaller scripts if that would help...?)
no, i'm after "less overall" rather than "smaller chunks". stuff like
library probes and the like are fine, because that's just hard-coded
for me anyway.
Post by Rob Landley
Rob
Rob Landley
2018-10-05 21:51:33 UTC
Permalink
Post by enh
Post by Rob Landley
In terms of eliminating unnecessary build plumbing, the tagged array stuff
(scripts/mktags.c) is the other big chunk of preprocessing, which is really
there to make a combined array/enum construct where you have a string name for
each entry _and_ a symbol of the same name resolving to the index of that entry
in the array. I would LOVE a way to do this with macros instead. (I can
stringify, but how do I get an increasing symbol number that goes up by one each
time it's referenced? I'd need INCREMENT(previous) to nest +1+1+1+1 and it's not
obvious how, which is prsumably why gcc has a __COUNTER__ macro, which is fiddly
and which I dunno if llvm implements it...
i'm kind of resigned to having to explicitly build the help, flags,
and arrays binaries and run them, and they all seem like they're
pulling their weight anyway. i just don't want to encourage too much
random other stuff, because i need to replicate all your shell scripts
in soong.
What does "replicating in soong" mean here? [googles...]

https://android.googlesource.com/platform/build/soong/

"The build logic is written in go using the blueprint framework."

You're using go as your build scripting language?
Post by enh
Post by Rob Landley
(I can also break up make.sh into multiple smaller scripts if that would help...?)
no, i'm after "less overall" rather than "smaller chunks". stuff like
library probes and the like are fine, because that's just hard-coded
for me anyway.
I was thinking it might help triage what's there and evaulate what's necessary.

I'm all for removing unecessary stuff, on general principles. Unfortunately,
when I wrote plumbing it's usually because I couldn't figure out how to get away
with not doing it.
Post by enh
Post by Rob Landley
Rob
Rob
enh
2018-10-05 23:39:43 UTC
Permalink
Post by Rob Landley
Post by enh
Post by Rob Landley
In terms of eliminating unnecessary build plumbing, the tagged array stuff
(scripts/mktags.c) is the other big chunk of preprocessing, which is really
there to make a combined array/enum construct where you have a string name for
each entry _and_ a symbol of the same name resolving to the index of that entry
in the array. I would LOVE a way to do this with macros instead. (I can
stringify, but how do I get an increasing symbol number that goes up by one each
time it's referenced? I'd need INCREMENT(previous) to nest +1+1+1+1 and it's not
obvious how, which is prsumably why gcc has a __COUNTER__ macro, which is fiddly
and which I dunno if llvm implements it...
i'm kind of resigned to having to explicitly build the help, flags,
and arrays binaries and run them, and they all seem like they're
pulling their weight anyway. i just don't want to encourage too much
random other stuff, because i need to replicate all your shell scripts
in soong.
What does "replicating in soong" mean here? [googles...]
https://android.googlesource.com/platform/build/soong/
"The build logic is written in go using the blueprint framework."
You're using go as your build scripting language?
hopefully not. there are genrules, but they require us to be honest
and specific about what's going on.
Post by Rob Landley
Post by enh
Post by Rob Landley
(I can also break up make.sh into multiple smaller scripts if that would help...?)
no, i'm after "less overall" rather than "smaller chunks". stuff like
library probes and the like are fine, because that's just hard-coded
for me anyway.
I was thinking it might help triage what's there and evaulate what's necessary.
thinking about it more, breaking it down into smaller scripts probably
would help.
Post by Rob Landley
I'm all for removing unecessary stuff, on general principles. Unfortunately,
when I wrote plumbing it's usually because I couldn't figure out how to get away
with not doing it.
well, for the specific case of getconf it turns out that reality is
far uglier than either of us could have predicted, so automation
wasn't likely to help us anyway. sadly whoever went first didn't
automate, so there's all kinds of random nonsense for us to copy. (see
other patch.)
Post by Rob Landley
Post by enh
Post by Rob Landley
Rob
Rob
Rob Landley
2018-10-04 20:23:55 UTC
Permalink
As someone who has a vested interest in having less generated code (because I'd
like to stop checking in generated files and use toybox in the AOSP build[1])...
do we even need to do this? How many of these defines are missing on
glibc/musl/bionic? If it's only one or two as I assume, could we just have the
#ifdefs instead?
Or are there actually a large number of these missing?
Years ago the linux kernel had clever config_enabled(CONFIG_BLAH) macros that
did a trick with commas and varargs to turn undefined symbols into a default
argument, and I've been meaning to try to extend that into CFG(THINGY) and
USE(THINGY, xxx) macros that don't require a preprocessed #include file.

Unfortunately they seem to have renamed it (broke my "apply DEVTMPFS_MOUNT to
initramfs" patch, that did), lemme see if I can dig that up... it's in
include/linux/kconfig.h, and consists of the following 5 macros plus explanation
comment:

#define __ARG_PLACEHOLDER_1 0,
#define __take_second_arg(__ignored, val, ...) val
/*
* Getting something that works in C and CPP for an arg that may or may
* not be defined is tricky. Here, if we have "#define CONFIG_BOOGER 1"
* we match on the placeholder define, insert the "0," for arg1 and generate
* the triplet (0, 1, 0). Then the last step cherry picks the 2nd arg (a one).
* When CONFIG_BOOGER is not defined, we generate a (... 1, 0) pair, and when
* the last step cherry picks the 2nd arg, we get a zero.
*/
#define __is_defined(x) ___is_defined(x)
#define ___is_defined(val) ____is_defined(__ARG_PLACEHOLDER_##val)
#define ____is_defined(arg1_or_junk) __take_second_arg(arg1_or_junk 1, 0)

So __is_defined(BLAH) resolves to 0 or 1 depending on whether BLAH is defined to
1 or not, using just macro expansion...

(The "need for multiple macros that call each other" is a thing I hit long ago
and described at https://landley.net/notes-2007.html#05-12-2007 .)

The problem is, we can't guarantee the macro is defined to 1, so I need to
invert the logic and have it be "does not resolve exactly to the macro name".
And ideally, we want:

ISDEFAULT(MACRONAME,default value)

Where it resolves to whatever the macro is defined to, else resolves to the
second argument...

Rob
Loading...