[yocto] [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools

[Josef Holzmayr]

Hi Trevor,

On 10.03.2017 21:49, Trevor Woerner wrote:
> Although the trend is to not care about licensing, I believe it is vitally
> important that we do our best to keep track of all the licensing from every
> package that is pulled into an image. If we're pulling in >1000 npm packages
> just for one node app, then that means we should have >1000 item list of each
> dependency and their respective licenses. Although it makes a recipe look
> ugly, I wouldn't want to drop this functionality due to aesthetic concerns.
> Maybe the license list could be moved to another file that is required by the
> "main" recipe file? Maybe the list could be moved to the bottom of the file?

Boiling that down, it sounds to me like the approach is the following:
1) Let the sub-package manager do its work as its meant to be.
2) If the sub-package manager supports version lockdown/shrinkwrapping. 
it shall be used.
3) The OE build process is only meant to take care of licensing.

(This could basically be seen as an additional Option 0 to the mail from 
yesterday: License-only recipes. [1])

Sounds like an interesting option indeed. Keeping it in the recipe 
means, in an abstract manner, that we need support for sub-licensing. 
Might be a viable route to go .

> In the case of node specifically, I don't think trying to create and maintain
> separate recipes for each and every dependency one might find in the npm
> registry would be a sane approach. Currently we embed the version info into
> the recipe filename. This will simply not scale to millions of npm packages,
> each with numerous versions.

It will not scale for human inspection. For metadata that is 
algorithmically generated and used, I personally don't think the sheer 
number is a killer argument.

> I've been playing with node a fair amount lately as it relates to OE and I
> have to say I've been quite impressed! These aren't easy things and I think
> there's a lot of good work happening.

Totally agreed. But implicitly, we tend to see npm as the reference for 
such sub-package managers. Is this a good way? Alexanders approach was 
to find a concept that fits all such constructions. Maybe its also 
worthwhile to think along the opposite lines: Treat each and every of 
those sub-package managers completely on its own, with all its 
specialities? (And hope that their number stays low....)

> Other than these (short-term?) issues devtool seems to be on the right
> track (?) It does, for example, generate a lockdown.json file and an
> npm-shrinkwrap.json file automatically. All we need is the package.json from
> the app developer, and that can be auto-generated via npm. I think we have to
> accept that node developers are going to want to develop on the target device
> itself, and when they're done they can hand us the package.json file which we
> can run devtool on which will generate the recipe for us.

I'm not too convinced that this is a good way. Especially when it comes 
to node modules that contain some native code, this becomes very ugly. 
My experience is that auto-processing that stuff adds megabytes of 
clutter, while all that matters in the end is a binary that is a couple 
if kilobytes. So how would one tackle that? Carve that out as a separate 
recipe again?

> As a short-term work-around, I've simply been creating an image with node+npm,
> running it on the device, copying over the package.json file, running npm
> install against it, then collecting up all the extra stuff that gets added
> to my image (as a result), and bundling all that into a platform-specific
> "bin_package" (bbclass). It works, but it's a multi-step process. If I could
> cut out some of those steps (once things from [1] are fixed), it would be an
> improvement.

Yeah, thats an option. I am rather providing custom compile and install 
stages, as the applications I'm working on have similar requisites, but 
I didn't want to go multi-step/binary.

Greetz

PS: being tired of typing sub-package manager again and again, how shall 
we call this? SPM? Application Package Managers (APM)?

[1] 
http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000489.html

-- 
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———————————————
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548

_____________________________________________________________
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548