[yocto] Questions on HOSTTOOLS and the documented host packages in the manual

[Gunnar Andersson]

Hi everyone

To get repeatable quality builds we ensure the build agents in CI are
identical and that they install a _minimal_ list of packages taken from the
instructions in the Yocto manual [1].  But from time to time some dependency
is missing, especially since things change in Yocto over time (e.g.
gzipnative class, and such).   So we were even missing gzip in one build
failure.  Now I couldn't confirm that the gzip binary is indeed missing - it
seems to be on our standard image, but possibly the configure step was
looking for some -devel package, headers or whatnot.

Still we have had our share of "works on my machine" answers from users that
don't use a controlled, docker-container like that of the CI build agents.

Sidestepping, since I anticipate this in your response:  I know there's
continuous work on repeatability, and tried to read what I can in the manual
on the new HOSTTOOL variable.  The mechanics of $PATH and softlinking the
binaries are clear, but I think I missed the intended usage and process
around it.   First, is it expected/recommended that each layer modifies this
variable?  If so, would layers add new host dependencies, or is the
intention to have layers further limit this with its own whitelist but still
based on a predefined set coming from the Poky layer?

I get that technically speaking you build in a lot of freedom, but from a
process perspective I think if the Yocto project defines a minimal set of
required host tools, then a Yocto-based project (like GDP) could take the
stance that all other dependencies must come from DEPENDS declaration on a
-native package.   Unless there is a clear agreed baseline, it's difficult
to file a bug on a recipe with respect to their -native dependencies.

Please confirm, is the manual (I refer to the Mega Manual [1]) intended to
be such a canonical list of host dependencies?

But if so:  Is it intentional or a bug that gzip (and bzip2, perhaps others)
is listed to be installed for Fedora, but not listed to be installed for
Ubuntu/Debian, see [1]?   Maybe the idea was that some things are always
there by default on those distros, but that seems brittle.  Shouldn't every
package be listed very explicitly?

Appreciate your comments...

- Gunnar

[1] http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#packages