[yocto] [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Paul Barker]

On Thu, 16 Mar 2017 09:17:57 +0100
Josef Holzmayr <holzmayr at rsi-elektrotechnik.de> wrote:

> I'll give it a shot and try to sum up the current state of affairs in 
> this discussion. In accordance to the "Package managers all the way 
> down"-presentation, I'm gonna use the term LPM (for language package 
> manager) for now on.
> 
> *Requirements*
> - repeatable and verifyable licensing of all dependencies that a LPM 
> pulls in.
> - locking down specific versions of packages and their dependencies for 
> reproductible builds.

Agreed with these.

> 
> *Optionals*
> - strict separation of fetch, compile, install stages. If a specifc LPM 
> requires it, those might be intermingled or left out [Mark]

I think what we need is strict separation of fetch (allowed to access
network) from the later tasks in a build (not allowed to access
networks). We need to continue to support users behind corporate
firewalls via mirroring and offline builds.

> - opaque packaging: similar to static linking, we should at least have a 
> way to bundle up a complete application into a single package. Maybe it 
> might even be the default (like rust does it at the moment).
> - leverage as much as possible of the functionality the LPMs provide 
> instead of reimplementing it.

We should also aim for automatic generation of recipes from pypi,
npmjs, crates.io, etc using devtool. I'm not sure how much of this has
already been implemented, I'm not entirely up-to-date on devtool
functionality.

Staying with NPM as the example, say we have a tree of 1000
dependencies for a top-level application recipe. I understand that
generating and then parsing 1000 recipes will bloat things and slow
down the build. However, we do need some way to apply patches to those
dependencies or fix metadata. For example, what if the license is
stated wrong on npmjs? What if the dependency list itself is wrong? What
I've always liked about OE is that you can fix this sort of problem in a
recipe when you need to.

> 
> *Wish List*
> - separating out the LPM infrastructure into one or more distinct 
> layers, not treating it as OE/bitbake core functionality. [Paul]
> - support for the use of multiple languages/LPMs inside a single recipe, 
> hopefully even package. [myself]
> 
> *Proposed Solutions*
> - having lockdown files shipped with the recipes (in whatever form to be 
> defined)
> - leveraging the recipe system to resolve licensing.
> 
> If we can boil things down to the common set that we all expect, it will 
> in my opnion serve as a blueprint for the actual implementation to follow.
> 
> Greetz,

Thanks,
Paul