[yocto] [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools

[Mark Hatle]

On 3/10/17 8:58 AM, Otavio Salvador wrote:
> On Fri, Mar 10, 2017 at 11:48 AM, Alexander Kanavin
> <alexander.kanavin at linux.intel.com> wrote:
>> On 03/10/2017 04:30 PM, Otavio Salvador wrote:
>>>
>>> When integrating the CHICKEN Scheme support onto the Yocto Project we
>>> dealt it using their installation tool but making the package of
>>> individual packages (eggs, in this specific case) as individual
>>> recipes. We went further and automated the recipe generation and this
>>> made it quite easy to maintain in long term.
>>>
>>> Take a look at:
>>>
>>> https://github.com/OSSystems/meta-chicken
>>
>>
>> Thanks, even though my Scheme-fu isn't great :)
>>
>> I'd like to avoid generating entire separate recipes though, because that
>> implies your custom-written tool would be figuring out where the dependency
>> source came from in the first place, and what are its own dependencies, when
>> creating the recipe, which can be tricky, breakage-prone guesswork.
> 
> In fact not; as you generate the recipes for the dependencies, it goes
> recursively and is always good.
> 
>> I want to use existing tools (like 'npm install') for getting the stuff from
>> the network - we don't really need full recipes, we just want to know the
>> licenses of the dependencies, and, if possible, lock them down to a specific
>> version.
> 
> Well we initially thought this would suffice but consider a security
> flaw. As many apps may be using different versions of same package it
> becomes a nightmare to figure which ones are affected. If using
> dependencies it is fine, for free.
> 

I'm wondering if there may be any way to generate these recipes on the fly (say
during ConfigParsed -- and then issue a reparse if things have changed?)

I've got concerns with this "new model" of development specifically around:

*) Bug fixes
   -) Security bugs especially

   When something is updated to fix a problem, how do we know we "got
everything"?  Something needs to flag the system to make it clear that not only
the items itself, but all of the things it depends on are at the correct
(revised) versions.

   The other problem is 'certified' devices.  Various certification requirements
don't allow just 'downloading new content whenever', but they often allow small
targeted fixes.  So in these cases, the user would likely want to know there was
an update for a specific reason and then 'backport, and not change the version'
to address their certification issues.  (And yes, I know of companies that used
to lock down version numbers of code, and simply upgrade anyway to get around
this -- it's always possible, but not exactly within the spirit of the
certification process..  targeted changes are much more reasonable in that
environment.)


*) License considerations
   A big problem in all of these new developer centric models is that they are
designed for back-end service were the license doesn't matter (as much).  For
embedded devices, we have to have a clear lineage of the license, code and
source (SRC_URI) to have any chance to "doing the right thing" or at worst
(defending work during a lawsuit).


*) Export/import control
    Unlike random open source projects, when you build a physical device there
may export/import control that has to happen as well.  Without the discrete
components being clearly visible to the developers -- it is -very- difficult to
satisfy the constraints of the various export/import requirements around the world.


I think auto-generating recipe contents help with these.  Having a good process
behind the auto-generation can make the license and software "update" procedure
a lot cleaner and provide the tracking needed for the export/import control.

I do see we need a better model for these components as we move away from the
core operating system and 'up the stack'.... better auto-generation and using
the tooling provided, while yet conforming to the needs we have makes the most
sense to me.  (The problem of course is most of these seem to need an expert, or
at least a highly interested developer to have a shot at working or keep working
over time.... the developer needs to be able to explain for the OE the
alternative model needed so we can collectively figure out how to do it best.)

I think this is a good start at the discussion, but other then listing concerns,
unfortunately I don't have any solutions to offer.

--Mark