[yocto] Supporting MEAN framework in Yocto

[Alexander Kanavin]

Hello,

I've done some research around MEAN stack and how Yocto could support 
it, and here are the findings.

*What is MEAN*

MEAN (http://mean.io/) is a Javascript framework for writing web 
applications with a client-server architecture. It's comprised of four 
major components:

- Node.js, the standalone Javascript interpreter that does not need a 
browser to run
- MongoDB, a C++ NoSQL database server
- Angular.js, a javascript application framework
- Express.js, another javascript application framework (not sure how the 
two interact - I'm not a JS specialist)

The major selling point of MEAN is that it is a 'full stack'. This means 
that both client and server side code is written using the same 
language, and shares many of the libraries. This is in contrast to the 
traditional LAMP stack, where server code is written in P* (Python, Perl 
or PHP) and client code is written in javascript, and presumably hiring 
developers with the right skillset is harder in such circumstances.

Starting a project in MEAN is very easy - the (slightly exaggerated) 
example from the website frontpage is:

$ sudo npm install -g mean-cli
$ mean init yourNewApp

*Why does Yocto need to support this?*

At the moment, it really does not. But we need to move with the times; 
if Yocto users want to put a MEAN app on their devices, Yocto should not 
get in their way, or declare that it isn't supported. There is a trend 
towards using node.js on the server side, and it will probably trickle 
over into embedded space as well.

*What are the difficulties?*

Placing node.js and Mongodb into images is not complicated. We have 
recipes for both (meta-oe has an outdated version of node.js, but 
there's also meta-node layer, which is totally up to date with 
upstream). The complications arise from the way javascript world handles 
project dependencies (see my previous email for a deeper look into 
this). It works like this:

- after initializing the project like above, you need to change into the 
project directory and do this:

$ npm install

- this recursively pulls all of the dependencies (including angular.js 
and express.js) into the node_modules directory of the project tree. 
Developers don't need to know or care what they are, and where they come 
from (and it's implied that licensing is not a worry either); the 
resulting set of libraries is bundled with the application and is not 
shared with other applications. As long as npm install completes without 
an error, that's all you need to worry about.

- unfortunately for us, the list of dependencies is truly enormous. The 
bare-bones MEAN app pulls in 1087 dependencies - and each of them has 
their own license, version, and upstream repository:

$ ls myNewApp/node_modules/|wc -w
1087

*What do we do with the 1087 dependencies?*

There is support in bitbake/devtool for dealing with npm-based projects 
- I didn't have time to try it (and it's also at the moment broken for 
the master branch due to rss), but it would probably not handle this 
very well:

- it reimplements 'npm install' as a custom fetcher and dependency 
resolver. I believe this is not the right way in the long run: we should 
let 'npm install' do its job, and inspect the tree after it's done to 
get the information we really need.

- the license information for all of the dependencies is written into 
the recipe, which makes it unnecessarily large. It is not clear how that 
information should be updated when needed.

- also, the dependencies are locked down through a separate lockdown 
file created by devtool, but it's not clear what to do when the lockdown 
file doesn't match reality, or when it otherwise needs to be updated. 
There should be a common, documented mechanism for all of this - see my 
previous email :)


Thanks,
Alex