[yocto] [npm] duplicate code

[Jean-marie Lemetayer]

Hi Stefan,

I thought about your idea of using Yocto to manage NPM package dependencies and I ran into an issue. NPM projects can have multiple dependencies on a single package, and sometimes with multiple versions. NPM will manage this by creating sub 'node_modules' tree.

Here is an example with a newly created angular application. The app depends on 3 different version of 'ansi-regex'. NPM will install the packages this way:
node_modules/ansi-regex @ 2.1.1
node_modules/cliui/node_modules/ansi-regex @ 3.0.0
node_modules/inquirer/node_modules/ansi-regex @ 4.1.0
node_modules/string-width/node_modules/ansi-regex @ 3.0.0
node_modules/@angular/compiler-cli/node_modules/ansi-regex @ 4.1.0

How can you handle that with Yocto ? I am not sure but I think it is not possible.

For that reason I think I will base my developments on the master branch.

----- Mail original -----
> De: "Stefan Herbrechtsmeier" <stefan at herbrechtsmeier.net>
> À: "Jean-marie Lemetayer" <jean-marie.lemetayer at savoirfairelinux.com>
> Cc: "Yocto-mailing-list" <yocto at yoctoproject.org>, "Savoir-faire Linux Rennes" <rennes at savoirfairelinux.com>
> Envoyé: Vendredi 4 Octobre 2019 16:55:47
> Objet: Re: [yocto] [npm] duplicate code

> Hi Jean-Marie,
> 
> Am 04.10.19 um 14:37 schrieb Jean-marie Lemetayer:
>> > I have recently worked on a yocto project using npm and I have seen some issues.
>> > I have solved a few but only for bitbake:
>> https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=eecba41822e86b69ebdb9cbc8fbfd512ad9a47d7
>> https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a34d0d539e5fdf341541fb628652d22289e80512
>> https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=e90cd2ed61b93ee7e290e7e592f1f0242ab5c281
>> https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a35abe31dc23916fd06afdb3a5e22a81ef3df579
>> 
>> As I have more time now, I wanted to continue my work by fixing devtool /
>> recipetool.
>> 
>> I have also checked the bugzilla for issues that I could fix / that should be
>> tested again:
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=10515
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=10760
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=11028
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=11029
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12534
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13415
>> 
>> I ended up with this todo list:
>>   - merge the duplicate code between bitbake and recipetool
>>   - fix the npm package name handling in recipetool
>>   - fix the npm package version handling in recipetool
>>   - fix the lockdown.json file generation in recipetool
>>   - create an example nodejs application to test all these cases
>>   - update the wiki using this example application:
>>     https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM
>> 
>> Finally, in my recent project, we were using angular (https://angular.io) for
>> the front-end. I have planned to add the work done to support angular
>> applications in yocto (angular.bbclass) and update recipetool to handle them.
> 
> Do you want to build an angular application? In this case you have to
> produce a machine (independent) package via a native npm build because
> you need to run the angular compiler on the build machine.

Yes, I have an 'angular-cli_x.y.z.bb' and an 'angular.bbclass' which depends on 'angular-cli-native'. I have planned to upstream this work.

> 
>> 
>> Your work sounds very interesting. The good point is that npm-shrinkwrap.json
>> and lockdown.json files (which have generation issues btw) will no longer be
>> required. But projects using npm can have a lot of dependencies (e.g. the
>> angular example app have 1053 dependencies).
> 
> A lot of dependencies are the problem at the moment. But many
> dependencies are not update frequently and my current solution assume
> that I can always use the latest version per major release version
> 
>> Is recipetool will be handling the whole recipes creation in one time ?
> 
> That's the problem at the moment. I don't manage to build multiple
> recipes from one recipetool run. Therefore I have a two stage approach.
> I use recipetool to build exact one recipe. A second script generates a
> list of all dependencies and call recipetool per package if the recipe
> doesn't already exist. This works but the run time is very high.
> 
>> Is it possible to see your work ? A public fork would be nice. I would gladly
>> help you / test your work / add my fixes if needed.
> 
> I only have a unfinished Proof of Concept. If you like I could clean it
> up a bit and post it on github as a separate layer.

Like I said at the start of this email, I will not base my dev on yours. But I am still interested by seeing your work.

Best regards,
Jean-Marie