Discussion:
Out of tree build support
(too old to reply)
Patrick Oppenlander
2017-12-18 21:57:13 UTC
Permalink
Hi Rob,

are you open to accepting patches for building out of tree?

I am working on a project where Toybox will be part of an automated build system, and this would make life a lot easier.

Patrick
Rob Landley
2017-12-18 23:41:28 UTC
Permalink
Post by Patrick Oppenlander
Hi Rob,
are you open to accepting patches for building out of tree?
I am working on a project where Toybox will be part of an automated
build system, and this would make life a lot easier.
Patrick
It's one of my todo list items, but there's backstory:

I've tried to make it so scripts/make.sh can be run separately (not via
the Makefile, that should be an essentially cosmetic wrapper around the
script-based build infrastructure). (In theory this means you can build
it on systems that haven't got a functional "make" yet. I have another
todo item to have a scripts/defconfig that populates generated/*.sh with
absolutely minimal dependencies, working even with apple's broken sed...)

I also have a "configure" file where you can persistently set default
values for overrideable build options. This is in shell syntax rather
than Makefile syntax so only the shell scripts import it, if I import it
from the Makefile it gets confused by the "[ -z "$BLAH" ] && BLAH=" syntax.

The problem is both the Makefile and scripts/make.sh use the generated/
directory, but the configure file isn't in Makefile syntax. I could put
the default value in two places but having the same piece of information
in two places pretty much guarantees they'll get out of sync eventually...

The HOSTCC variable already has this problem. The reason I haven't
implemented this yet is every time I sit down to implment it I get
distracted by trying to come up with a better design for this...

(Probably configure should have VALUE?="blah" and then a shell wrapper
that parses and sets that syntax.)

Rob

(The build has three conflicting use cases, which affect the
environmental dependencies and expected command line user interface. 1)
Conventional "configure/make/install" using gmake. 2) Portable build not
depending on anything toybox doesn't itself provide (building on BSD and
such). And android's doing ninja, which I should have support for but
ninja is a moving target, the one 14.4 installs doesn't work at all for
android...)
Rob Landley
2017-12-19 01:04:58 UTC
Permalink
Post by Rob Landley
(Probably configure should have VALUE?="blah" and then a shell wrapper
that parses and sets that syntax.)
Except when I sat down to try to implement that (again) I hit the
problem that configure variable defautls reference other config
variables (such as CROSS_COMPILE="$CROSS") and even writing a simple
parser to find ?= lines and turn them into = lines, to import that as
makefile syntax it has to be CROSS_COMPILE="$(CROSS)" with gratuitous
parentheses, and that a giant rathole.

Having variables you can set in both shell and makefile contexts turns
out to be kind of annoying. What I need to do is remove all the
remaining logic from the makefile and put it in the script and have the
makefile be just a dumb wrapper around the scripts like it was intended
to be...

Rob
Patrick Oppenlander
2017-12-19 01:49:45 UTC
Permalink
Post by Rob Landley
Post by Patrick Oppenlander
Hi Rob,
are you open to accepting patches for building out of tree?
I am working on a project where Toybox will be part of an automated
build system, and this would make life a lot easier.
Patrick
I've tried to make it so scripts/make.sh can be run separately (not via
the Makefile, that should be an essentially cosmetic wrapper around the
script-based build infrastructure). (In theory this means you can build
it on systems that haven't got a functional "make" yet. I have another
todo item to have a scripts/defconfig that populates generated/*.sh with
absolutely minimal dependencies, working even with apple's broken sed...)
That's a nasty chicken-and-egg problem indeed. I just thought you hated make :)
Post by Rob Landley
I also have a "configure" file where you can persistently set default
values for overrideable build options. This is in shell syntax rather
than Makefile syntax so only the shell scripts import it, if I import it
from the Makefile it gets confused by the "[ -z "$BLAH" ] && BLAH=" syntax.
I have another project which has the same problem, but it also needs the variables available for 'ld' to use in link scripts. The only workable solution I know of is to have a config file which is processed by a script to generate config.mk, config.ld, config.sh and config.h which can be included as required.

I may be able to hack something together at some point in the future if you're at all interested in this idea.
Post by Rob Landley
(The build has three conflicting use cases, which affect the
environmental dependencies and expected command line user interface. 1)
Conventional "configure/make/install" using gmake. 2) Portable build not
depending on anything toybox doesn't itself provide (building on BSD and
such). And android's doing ninja, which I should have support for but
ninja is a moving target, the one 14.4 installs doesn't work at all for
android...)
Nasty indeed.

I have done some hacking on the scripts. This currently works:

mkdir /tmp/toybox_build
make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile defconfig
make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile

So does this:

mkdir /tmp/toybox_build
make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile defconfig
cd /tmp/toybox_build && ~/src/ext/toybox/scripts/make.sh

So do other targets (like tests, etc). bloatcheck looks to be broken on trunk at the moment (incorrect dep on toybox_old?).

How do you intend to do the defconfig step using scripts?

I've pushed the current state of affairs to https://github.com/pattop/toybox if you're interested in taking a look. The symlinks in kconfig are a bit nasty (they really should go in the build directory to avoid polluting the source dir, but as they're always the same...). I got this working with vpaths earlier but wasn't too thrilled with the result.

CC'ing you as I don't seem to be getting list emails at the moment.

Patrick
Rob Landley
2017-12-19 22:24:53 UTC
Permalink
Post by Patrick Oppenlander
That's a nasty chicken-and-egg problem indeed. I just thought you hated make :)
Oh I do. But I tend to have programming opinions for a reason. :)
Post by Patrick Oppenlander
Post by Rob Landley
I also have a "configure" file where you can persistently set default
values for overrideable build options. This is in shell syntax rather
than Makefile syntax so only the shell scripts import it, if I import it
from the Makefile it gets confused by the "[ -z "$BLAH" ] && BLAH=" syntax.
I have another project which has the same problem, but it also needs the
variables available for 'ld' to use in link scripts. The only workable
solution I know of is to have a config file which is processed by a
script to generate config.mk, config.ld, config.sh and config.h which
can be included as required.
Which is more build dependencies, and the parsing gets nontrivial.

Really what I need to do is come up with a way to build toybox sed and
toysh standalone from a canned script with no config step, and then
build the rest with that.

(See also "make install_airlock".)

Speaking of which, did I ever point you at the cp -s trick?

$ cp -rs "$PWD"/toybox walrus
$ cd walrus
$ make distclean && make defconfig && make

(I taught toybox to work with relative rather than absolute paths. The
gnu/dammit version complains about relative paths, and yes that means
even "cp -rs toybox walrus" won't work unless you're using toybox cp.
It's because they don't bother to calculate the dynamic ../ stack.)
Post by Patrick Oppenlander
mkdir /tmp/toybox_build
make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile defconfig
make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile
mkdir /tmp/toybox_build
make -C /tmp/toybox_build -f ~/src/ext/toybox/Makefile defconfig
cd /tmp/toybox_build && ~/src/ext/toybox/scripts/make.sh
So do other targets (like tests, etc). bloatcheck looks to be broken on
trunk at the moment (incorrect dep on toybox_old?).
Blah, "make baseline" is putting it in generated/unstripped but it looks
like "make bloatcheck" is still expecting it at the top level. [fixed]
Post by Patrick Oppenlander
How do you intend to do the defconfig step using scripts?
One of my todo items is writing a new kconfig to replace the kernel
plumbing I borrowed way back when. (Which is from 2.6.12.)
Post by Patrick Oppenlander
I've pushed the current state of affairs to
https://github.com/pattop/toybox if you're interested in taking a look.
Hmmmm... I see what you're doing, but you're adding complexity to the
Makefile and I want to strip it out and move it into the scripts.
Post by Patrick Oppenlander
The symlinks in kconfig are a bit nasty (they really should go in the
build directory to avoid polluting the source dir, but as they're always
the same...). I got this working with vpaths earlier but wasn't too
thrilled with the result.
CC'ing you as I don't seem to be getting list emails at the moment.
I seem to be, but I think I've been cc'd on everything the past couple
days anyway?

Rob
Patrick Oppenlander
2017-12-20 00:25:06 UTC
Permalink
Post by Rob Landley
Really what I need to do is come up with a way to build toybox sed and
toysh standalone from a canned script with no config step, and then
build the rest with that.
Makes sense.
Post by Rob Landley
Speaking of which, did I ever point you at the cp -s trick?
$ cp -rs "$PWD"/toybox walrus
$ cd walrus
$ make distclean && make defconfig && make
(I taught toybox to work with relative rather than absolute paths. The
gnu/dammit version complains about relative paths, and yes that means
even "cp -rs toybox walrus" won't work unless you're using toybox cp.
It's because they don't bother to calculate the dynamic ../ stack.)
That is a neat trick. First time I've seen it, thanks!
Post by Rob Landley
One of my todo items is writing a new kconfig to replace the kernel
plumbing I borrowed way back when. (Which is from 2.6.12.)
Even before that the issue is that there's no way to build kconfig from a script, so the *config targets depend on make.
Post by Rob Landley
Post by Patrick Oppenlander
I've pushed the current state of affairs to
https://github.com/pattop/toybox if you're interested in taking a look.
Hmmmm... I see what you're doing, but you're adding complexity to the
Makefile and I want to strip it out and move it into the scripts.
I think the changes to the top level Makefile aren't too bad. They really just boil down to teaching make where to find sources & scripts by sprinkling in a few $(SRCDIR)s and don't change any of the logic at all.

kconfig/Makefile probably needs to be replaced by a script so the hackery in there is fairly moot. Would you be interested in a script that replaces this?

Are you happy with the direction of the changes to the scripts in general?

It might be worth considering passing $(MAKECMDGOALS) to make.sh (or a higher level driver script) at some point. It could then call single.sh or test.sh and eliminate the requirement for .singlemake. You could then call make.sh from a catch all rule in the top level makefile and really turn the makefile into a trivial wrapper.
Post by Rob Landley
Post by Patrick Oppenlander
The symlinks in kconfig are a bit nasty (they really should go in the
build directory to avoid polluting the source dir, but as they're always
the same...). I got this working with vpaths earlier but wasn't too
thrilled with the result.
What's the reason for the symlinks in kconfig anyway? I had a quick look through the history and couldn't find anything obvious.
Post by Rob Landley
Post by Patrick Oppenlander
CC'ing you as I don't seem to be getting list emails at the moment.
I seem to be, but I think I've been cc'd on everything the past couple
days anyway?
I'm not receiving emails for this thread. Maybe the mailing list is smart enough to know you're CC'ing me?

Patrick
Rob Landley
2017-12-21 17:16:06 UTC
Permalink
Post by Patrick Oppenlander
Post by Rob Landley
Really what I need to do is come up with a way to build toybox sed and
toysh standalone from a canned script with no config step, and then
build the rest with that.
Makes sense.
Post by Rob Landley
Speaking of which, did I ever point you at the cp -s trick?
   $ cp -rs "$PWD"/toybox walrus
   $ cd walrus
   $ make distclean && make defconfig && make
(I taught toybox to work with relative rather than absolute paths. The
gnu/dammit version complains about relative paths, and yes that means
even "cp -rs toybox walrus" won't work unless you're using toybox cp.
It's because they don't bother to calculate the dynamic ../ stack.)
That is a neat trick. First time I've seen it, thanks!
https://xkcd.com/1053/

(I keep thinking the fact I've repeated something 30 times over the
years means people are sick of it, when most of them have never heard of
it. One of the reasons I'm so interested in computer history is
preserving what used to be common knowledge, because
https://web.archive.org/web/20120111055334/http://wrttn.in/04af1a having
to come from archive.org sadly _isn't_ ironic, it's harmonic. It
_reinforces_ the darn point.)
Post by Patrick Oppenlander
Post by Rob Landley
One of my todo items is writing a new kconfig to replace the kernel
plumbing I borrowed way back when. (Which is from 2.6.12.)
Even before that the issue is that there's no way to build kconfig from
a script, so the *config targets depend on make.
There's no way to build the _current_ kconfig plumbing from a script.
But if I write new infrastructure...

(Alas it's another thing that's beyond the threshold of what I can sit
down and do in a couple hours from a standing start, which would be
negatively impacted by interruption and forced context switch...)
Post by Patrick Oppenlander
Post by Rob Landley
Post by Patrick Oppenlander
I've pushed the current state of affairs to
https://github.com/pattop/toybox if you're interested in taking a look.
Hmmmm... I see what you're doing, but you're adding complexity to the
Makefile and I want to strip it out and move it into the scripts.
I think the changes to the top level Makefile aren't too bad. They
really just boil down to teaching make where to find sources & scripts
by sprinkling in a few $(SRCDIR)s and don't change any of the logic at all.
Complexity often accumulates in small amounts. I'd prefer to solve this
by _removing_ stuff from Makefile instead of adding it.

Looking at the design, I suspect I should rename "scripts" to "make" and
have a subdirectory under there for files that aren't a build target
(make/include or something).

Then the top level Makefile should be a wrapper that _just_ calls
scripts under "make". For the moment kconfig/Makefile can stay crazy
because that whole mess is its own can of worms.

But I need to close tabs before opening a new one just now...
Post by Patrick Oppenlander
kconfig/Makefile probably needs to be replaced by a script so the
hackery in there is fairly moot. Would you be interested in a script
that replaces this?
Somebody wrote some plumbing to (partially) replace it in awk once, but
I just couldn't maintain it. :(
Post by Patrick Oppenlander
Are you happy with the direction of the changes to the scripts in general?
Sort of?

Make provides an expected command line API. The classic build is
"./configure ; make ; make install" and long ago I ranted (in two parts)
about how that's annoyingly obsolete at:


http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000859.html

http://lists.landley.net/pipermail/aboriginal-landley.net/2011-June/000860.html

But we haven't standardized on something new and better yet (the way cvs
gave way to git). Instead we've got fragmentation: the gnu guys did
their own "automake" syntax which is almost sane but nobody else picked
it up because it had gnu all over it and those were the guys who did
"info". Android decided to use something called ninja while
simultaneously switching to an llvm toolchain that wants to build with
cmake... Despite the many downsides of make, it has the advantages that
it's posix and still near-ubiquitous.

Getting back to configure/make/install: the Linux kernel swapped out
configure for "make defconfig" and friends, and as with Linux switching
to git lots of people are familiar with that now. It was copied by
busybox and uclibc, spread from there to buildroot (which started life
as uClibc's test harness), and now openembedded and so on have taken it
from there...

That said, we shouldn't be tied to a given implementation of either. An
implementation is not a standard, so you can't _have_ a standard until
there are two compatible implementations. (This was official IETF policy
at one point, dunno if it still is.)

So I want to write my own mostly-bash-compatible shell, I want to write
my own mostly-gmake-compatible make, and I want to write my own kconfig
plumbing.

Long ago I tried to push some of my kconfig changes from busybox
upstream into linux-kernel, this would have been maybe 2007? And it got
derailed into Matt Mackall and such bikeshedding about how if kconfig is
going to become generally useful for other projects it needs to become
its own separate package, and this blue sky plan was used as an excuse
not to take my patches unless I volunteered to do an order of magnitude
more work, and as usual I went "nuts to your white mice" and went back
to what I was doing because I was _already_ three interrupts deep from
what I was _trying_ to work on...

Anyway, the point is if I write a new menuconfig implementation that's
BSD licensed, there's a nonzero chance other projects might pick it up.
I only personally use menuconfig and the command line ones, so I'll
leave the qt/gtk arguments to somebody else (see "delegate to nobody" in
my standard Linux gui rant).

But again: too many open tabs just now.
Post by Patrick Oppenlander
It might be worth considering passing $(MAKECMDGOALS) to make.sh (or a
higher level driver script) at some point.
Then the layers have an unbounded API between them.
Post by Patrick Oppenlander
It could then call single.sh
or test.sh and eliminate the requirement for .singlemake.
You can already call "scripts/single.sh command", singlemake just
provides make targets for each command and its test_command. Having
make.sh be a dispatcher to other scripts on top of what it already does
is not an improvement.
Post by Patrick Oppenlander
You could then
call make.sh from a catch all rule in the top level makefile and really
turn the makefile into a trivial wrapper.
Not quite the direction I want to go in, thanks.
Post by Patrick Oppenlander
Post by Rob Landley
Post by Patrick Oppenlander
The symlinks in kconfig are a bit nasty (they really should go in the
build directory to avoid polluting the source dir, but as they're always
the same...). I got this working with vpaths earlier but wasn't too
thrilled with the result.
What's the reason for the symlinks in kconfig anyway? I had a quick look
through the history and couldn't find anything obvious.
Look at linux/scripts/kconfig in the kernel source. I made minimal
changes because at the time I cared about tracking upstream (and then
never did), and because I want to throw out that whole directory and
replace it with a fresh implementation in C.

I note that menuconfig, vi, and screen have similar plumbing. I
prototyped that plumbing a bit with hexedit and top (the
interestingtimes.c stuff which is my not-curses), but the guy part is
only half of it, the reading/writing/dependency resolution part is the
other half.

Which HAS BACKSTORY. (As everything I do these days seems to.)
Specifically, I tried to push some simple miniconfig support upstream
into the kernel in 2005:

https://lwn.net/Articles/160497/
https://lwn.net/Articles/161086/

Which turned into some dude named Roman Zippel bikeshedding at me to try
to get me to go solve some unrelated problem for him and do 10x as much
work as I'd done and he'd hold my patch hostage until then, so I went
away and tried again a year later:

http://lkml.iu.edu/hypermail/linux/kernel/0607.0/1805.html

And it was still bikeshedding, so I went off and did my own support for it:

http://landley.net/aboriginal/FAQ.html#dev_miniconfig
https://github.com/landley/aboriginal/blob/master/more/miniconfig.sh

And pretty much ignored the kernel guys. Meanwhile, the kernel grew some
crazy new stuff:

https://github.com/landley/aboriginal/blob/master/sources/patches/linux-deeplystupid.patch
https://github.com/landley/aboriginal/blob/master/sources/baseconfig-linux#L62

So I dug into the kconfig source a bit to see if a proper "make
miniconfig" mode would be easy to do (spoiler: nope), and it's been
"write a new one" ever since.

But I used some of that experience writing
https://github.com/landley/toybox/blob/master/scripts/config2help.c and
every time I poke at that file I get distracted by "I should write a
full kconfig replacement and make this a function of it", which is a
problem significantly larger in scope than what it does...
Post by Patrick Oppenlander
Post by Rob Landley
Post by Patrick Oppenlander
CC'ing you as I don't seem to be getting list emails at the moment.
I seem to be, but I think I've been cc'd on everything the past couple
days anyway?
I'm not receiving emails for this thread. Maybe the mailing list is
smart enough to know you're CC'ing me?
Ah, you're using gmail. This is a known perennial bug in gmail:

https://landley.net/notes-2011.html#05-04-2011

Which the gmail developers insist is not a bug, it's a feature:

https://landley.net/notes-2011.html#30-04-2011

But it's a feature that can't be worked around if you're using
Thunderbird due to conflicting hardwired assumptions:

https://landley.net/notes-2012.html#15-10-2012

And so on, and so forth. (Everything is backstory today. I should step
away from the computer.)

Rob
Rob Landley
2017-12-21 23:00:10 UTC
Permalink
Post by Rob Landley
Post by Patrick Oppenlander
I'm not receiving emails for this thread. Maybe the mailing list is
smart enough to know you're CC'ing me?
https://landley.net/notes-2011.html#05-04-2011
And of course as soon as I send that I notice that the OTHER known
perennial bug in gmail has trigered, namely gmail has false positived on
an email as spam, and when it refused delivery of that message for every
list recipient using gmail, and refused delivery of multiple retries
from dreamhost's list server, the list server disabled their list
subscriptions because mail was undeliverable.

Here's the one it refused:

http://lists.landley.net/pipermail/toybox-landley.net/2017-December/009309.html

And here's a represenative bounce message:

Final-Recipient: rfc822; soapboxcicero+***@gmail.com
Original-Recipient: rfc822;soapboxcicero+***@gmail.com
Action: failed
Status: 5.7.1
Remote-MTA: dns; gmail-smtp-in.l.google.com
Diagnostic-Code: smtp; 550-5.7.1 [64.90.62.195 18] Our system has
detected
that this message is 550-5.7.1 likely suspicious due to the very low
reputation of the sending IP 550-5.7.1 address. To best protect our
users
from spam, the message has been 550-5.7.1 blocked. Please visit 550
5.7.1
https://support.google.com/mail/answer/188131 for more information.
p76si13464492pfa.247 - gsmtp

It does this about once a month these days. I remember it did it while
it was in tokyo, but it looks like last time I complained about it here
was September:

http://lists.landley.net/pipermail/toybox-landley.net/2017-September/009149.html

So that's another reason why you might not have gotten all the emails. I
just finished manually re-enabling each user through the web ui (because
dreamhost won't give me command line access to the mailman plumbing), so
it should be back...

There's a reason I check the web archive so much...

Rob

Patrick Oppenlander
2017-12-20 01:01:24 UTC
Permalink
Post by Rob Landley
Really what I need to do is come up with a way to build toybox sed and
toysh...
How far are toybox sed & toysh from being usable to build toybox?

Patrick
Rob Landley
2017-12-20 19:47:22 UTC
Permalink
Post by Patrick Oppenlander
Post by Rob Landley
Really what I need to do is come up with a way to build toybox sed and
toysh...
How far are toybox sed & toysh from being usable to build toybox?
Sed's fine, I've built Linux From Scratch with it (albeit an older
version, 6.3 I think) and even done some patches for things like perl bugs:

https://github.com/landley/toybox/commit/32b3587af261

Which I found because I was building perl with it.

I haven't properly opened the toysh can of worms yet because toybox
isn't my dayjob and that one is going to need multiple consecutive
months to do right. Unfortunately most of the big remaining roadmap
items are still there because they require large consecutive chunks of
focus that the tiny little startup I've been working at since 2014 can't
afford to give me without going under.

Unfortunately I haven't had as much spare time the past year and a half
as I did before my startup ran out of money and went into survival mode.
(Which is as much exhausting as time consuming, I _do_ have spare time,
but I spend it sleeping. The real problem is I can never focus on
anything, I get high priority crisis crisis crisis world on fire
interruptions that derail what I'm working on and by the time I get back
it would be faster to start over. After about 30 of those I got a little
gun-shy about starting anything big. I've sat down and started over on
the dd cleanup _four_times_ now, and that's not even all _that_ big.)

That said, my 2013 talk on my grand plan for turning android into a
development environment is coming up on 5 years ago, and if I'm going to
make it happen I have to get unstuck pretty much now. (I'm looking at
selling my house. Last ELC Tim Bird asked what it would take to buy six
months of my time from my employer to do toysh, and I named him a
figure, and he went away and I never heard back. I think Sony had a
reorg or something. Sigh, I feel really bad about the rate of
development but I don't have as many gaps in my life to fit development
into as I used to. Downside of a dayjob at a startup that's _also_
trying to change the world...)

But hey, christmas break coming up. I'm hoping I can use it to make next
release less embarassingly "I got nothin'" than last release was. Maybe
I can get _one_ of my larger mostly finished things done. I called
getconf "mostly done" back in March, according to
http://landley.net/notes-2017.html#04-03-2017 anyway...

Speaking of which, I should catch up on my blog... there we go, another
month posted. (I didn't check twitter or my back email for things I
forgot to write about at the time, but eh. Edited to at least make the
tags match up. Next entry trails off half-finished and requires surgery,
of course...)

Rob

"I am sorry to have wearied you with so long a letter but I did not have
time to write you a short one" - Blaise Pascal
Continue reading on narkive:
Loading...