[yocto] Can Yocto treat layers like an external package?

[Koehler, Yannick (HPN Aruba)]

Hi Marcello,

Not sure if I understood what you meant but, I am using a single repo with submodule for vendor layers, as such, submodule kind of solve the "temptation" for our developer team to go play in there, since playing there is a minefields thanks to the complicated submodule process.  But, I do not want to rely on the difficulty of submodules to tell my dev not to go change the vendor layer, which is way I do not want them in my repo at all.

Note that private issues is a different matter and I am not addressing it here and there are multiple solution such as subtree, repo, submodules, upcoming GVFS, etc...

>It doesn't sound too far fetched, though. bitbake already contains most
>(all?) of the code to make this work. I think you would just need to come up with a reasonable UI to interact with that.

This is my opinion, I think we could alter bitbake to retrieve the SRC_URI and S information from the bblayers.conf (instead of having recipes like for package). 

    sample 
    # LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
    # changes incompatibly
    LCONF_VERSION = "6"

    BBPATH = "${TOPDIR}"
    BBFILES ?= ""

    BBLAYERS ?= " \
      ##OEROOT##/meta \
      ##OEROOT##/meta-yocto \
      ##OEROOT##/meta-yocto-bsp \
      ##OEROOT##/meta-openembedded;SRC_URI="git://git@github.com/openembedded/meta-openembedded;protocol=ssh;branch=master";SRCREV="${AUTOREV}" \
      "
    BBLAYERS_NON_REMOVABLE ?= " \
      ##OEROOT##/meta \
      ##OEROOT##/meta-yocto \
      "

Bitbake could then fetch/update the layer from the SRC_URI into the location given "##OEROOT##/meta-openembedded" prior to parsing.

--
Yannick Koehler

-----Original Message-----
From: Magallon, Marcelo 
Sent: Thursday, May 25, 2017 10:40 AM
To: Koehler, Yannick (HPN Aruba) <yannick.koehler at hpe.com>
Cc: Andrea Galbusera <gizero at gmail.com>; Trevor Woerner <twoerner at gmail.com>; yocto at yoctoproject.org
Subject: Re: [yocto] Can Yocto treat layers like an external package?

On Thu, May 25, 2017 at 01:13:03PM +0000, Koehler, Yannick (HPN Aruba) wrote:

> The case I am mostly interested about is the Vendor layers, if I pull 
> in meta-openembedded or meta-freescale in my yocto distro, I will 
> never touch those layer at the git level, instead whatever change I 
> want will be done in my private layer which is the main reason for 
> layer as I understand it (being able to change other’s vendor layer 
> without changing them).

A "solution" that we use is have a single repo with all the layers, and manage that using git subtrees.

I would strongly discourage other people from implementing such a solution, as it solved one problem (creating and updating a workspace with the correct layers is trivial), but it has created many others. The biggest one in my view is that people feel encouraged to make modifications to the vendor layers, which makes updating harder than it should. Having a single repo with layers within directories also makes it cumbersome to implement access controls (basically "you can commit to directories A, B and C, but not X, Y or Z"). It also encourages a kind of coupling that makes things more complicated in the long run.

> I fail to see why people would seek a non-bitbake solution such as 
> repo, submodules or others if bitbake was able to do that aspect.  If 
> there is a project already for doing this, I would be interested in 
> participating to its development and I may have one or two helper in 
> my team to help out on this.

My guess is that's because bitbake would have to read a file with the layer metadata, fetch the layers that you want, and then read the recipes. If the layers are handled just like recipes, (with SRC_URI, SRCREV, S, etc), bitbake needs to be able to read two different sets of recipes. That might work by changing BBFILES or BBPATH. At that point it the UI becomes a bit awkward, I think.

It doesn't sound too far fetched, though. bitbake already contains most
(all?) of the code to make this work. I think you would just need to come up with a reasonable UI to interact with that.

With our solution, getting updated layers (non-vendor) it's just a matter of "git pull". Solutions built around repo are equally simple.

> For private layers, I do understand and see why an external solution 
> to bitbake would be better, since bitbake will not offer support for 
> branch and change management which is normal as bitbake is only a 
> build tool, not a developer tool.

Well, if you obtain layers using recipes (or something very close to that), you could include branch information in SRC_URI, as usual.

Marcelo