[yocto] How to write a distro layer?

[Uwe Geuder]

Thanks Vincent for your quick reply!

On Fri, Aug 3, 2018 at 12:44 PM, Vincent Prince <vincent.prince.fr-at-...> wrote:
> 2018-08-03 10:39 GMT+02:00 Uwe Geuder <jrswdnan22 at snkmail.com>:

>>    One of the few informations I could find about how to create my own
>>    layer was something like
>>    https://www.openembedded.org/wiki/Creating_a_new_Layer

> Did you check
> https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#creating-a-general-layer-using-the-bitbake-layers-script

Yes. The template generated by that script is very minimal and
generic. I don't see any guidance for distro layers in particular.  As
a matter fact I believe our layer has been created by that script
(although that was long before my time).


>> The yocto reference manual says for many variables: "Set it in your
>> local.conf". For me they are policy, so why shouldn't I set them in
>> distro/<distro>.conf?

> Did you also check
> https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#creating-your-own-distribution

I had read that many times before, but not recently so let's check again...

| Note
| The DISTRO variable in your local.conf file determines the name of
| your distribution

[...]

| Point to Your distribution configuration file: In your local.conf
|  file in the Build Directory, set your DISTRO variable to point to
|  your distribution's configuration file. For example, if your
|  distribution's configuration file is named mydistro.conf, then you
|  point to it as follows:
|
|     DISTRO = "mydistro"
                    

Correct, that's what I remembered. But if I set it in local.conf
before adding my layer the bitbake sanity check will complain that
there is no such layer. And if I modify local.conf after adding the
layer all recipes will be re-parsed when building the first time.

I don't see any guidance in what order the steps should be carried out
when setting up a new build area.

They do indicate that you should copy another distro's conf file,
that's why I asked whether using "require" as we do is a bad/dangerous
idea.

> You can look some Freescale example here:
> https://github.com/Freescale/meta-freescale-distro

Good, I hadn't found those. At least the distro/<name>.conf files look
pretty much like I would expect them.

But all of them (I understand the layer defines 8 different distros)
do set the DISTRO variable. I haven't run the yocto-check-layer for a
while, but the last time I did it complained that a distro layer MUST
NOT set its own DISTRO variable. Do you happen to know whether
meta-freescale-distro passes the yocto-check-layer script without
complaints? (I know I could test myself...)

> In fact, when Yocto manual refers to adding something to local.conf,
> it is the minimal way to do it, but the cleanest is indeed to set it
> in machine/distro config file.

That has also been my thinking. If that's generally accepted we
probably should send a patch to the manual. It's pretty misleading if
the manual without any further explanations says "do it way A" if "way
B" often is actually the better one.


> 1.) DISTRO is local to build folder,

By "local to build folder", do you mean that within the same build folder
one cannot build more than 1 distro? Not relevant for me, because I
have only one for the time being, but just to understand the message.

> so it should be in local.conf, and you can use a wrapper script to help
> creating this local.conf file,

Yes, that's what we do. Without a wrapper script builds were unlikely
to be reproducible by any measure...


> https://github.com/Freescale/fsl-community-bsp-base/blob/master/setup-environment

Interesting. I cannot spot how that script handles layers.

My own script

1) sources oe-init-build-env
2) modifies the generated conf/local.conf
3) runs bitbake-layer add-layer for all layers we need in our distro

>> In an attempt to make our code more in line with what I understand to be
>> correct style, I tried to move all policy settings to
>> distro/<distro>.conf and the DISTRO setting to local.conf.
>> 
>> However that created a couple of problems:
>> 
>> 1. If I set DISTRO in local.conf before adding all needed layers, my own
>> distro layer last, bitbake-layers fails in a sanity check that the
>> distro does not exist. Fair enough, it doesn't know about before it has
>> been added.
>> 
>> 2. If I first modify my local.conf with all my settings except DISTRO,
>> then run all the add-layer operations and finally modify local.conf a
>> 2nd time to set DISTRO correctly the build works. However, all recipes
>> are reparsed when the build starts. I guess this is caused by the config
>> change.
>> 
>> So in the end having the DISTRO setting in layer.conf seems to work
>> best. All add-layer commands succeed and recipes can be used from the
>> cache when the build starts. Best, except that I don't think that's the
>> way it's documented.
>> 
>> Any thoughts what I might be missing?

> You can use git submodule, or google repo to create Yocto environment,

Yes, we have all layers as submodules in a single git repo, so we
always can do a clean build "from a single SHA1"

> you don't need to create your layer each time 

What do you mean by creating a layer? Of course the source is in
git and it only changes when necessary.

What I do in every new build area is run add-layer. (Currently still
repeatedly for one layer at a time, but I think I read in the new
release notes that bitbake-layer can now add several layers at once.)

I have thought about saving the generated layers.conf file and
restoring it by my setup script. But somehow I feel more confident if
the parsing is done for each new build area. Otherwise one might end
up in the situation that one day you note that a change made weeks or
months ago has broken something and nobody noted it.

> and you don't have to compile before using bitbake-layers.

Not sure what you mean by that. Of course I cannot build/compile
anything before the build area is set-up completely.

Bitbake-layer parses all recipes of the layer it adds. If I do not
modify local.conf after adding the last layer the first build
starts immediately from cached recipes.

If I modify local.conf to set DISTRO after adding the last layer, the
first build re-parses everything. Not that it's a huge increase to the
whole build time, but it does not feel right. 

Regards,

Uwe

---
Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
uwe.gexder at neuroeventlabs.com (Bot check: fix one obvious typo)