[linux-yocto] looking for doc for in-house linux-kernel dev tree

[Bruce Ashfield]

On 13-06-07 12:50 PM, Jean-François Dagenais wrote:
> Hi Bruce, thanks for the answer... here's a follow up.
>
> On 2013-06-04, at 10:55 AM, Bruce Ashfield wrote:
>
>> On 13-06-03 10:29 PM, Jean-François Dagenais wrote:
>>> Hi all,
>>>
>>> I've gone through lots of doc concerning kernel development but couldn't find the right fit.
>>
>> Are you working on the yocto master branches ? I ask, because there are
>> different features in different branches of the project.
>
> My current focus was to get things working on branch danny (poky, meta-intel,
> meta-oe). We are shortly moving ahead to dylan and kernel 3.8 though.

Some late night answers below.

>
>>
>>> As described in manuals, we've copy/pasted the kernel meta branch's crownbay bsp folder and customized it. Initially my reflex was to include the crownbay-standard.scc from my BSP's .scc file instead. This way my bsp would inherit crownbay improvements directly. Unfortunately I haven't found a way to disable kernel CONFIG_* flags which crownbay (or what it includes) explicitely enables. (Thoughts here?) So here we are. In recipe space, we've bbappended linux-yocto-3.4 so we can override the SRC_URI to use our in-office linux kernel git clone (or my development clone) populated with my merged branch and our customized meta branch, as well as provide different sha1's for the "machine" and "meta".
>>
>> If you look at poky-extra/meta-kernel-dev, you'll see that the approach
>> of pointing to a different repo is common, and supported. I do something
>> very similar for other kernel trees that I maintain, including overriding
>> and changing all sorts of configs.
>>
>> So that matches your first reaction and something that I've been
>> using for quite some time.
>
> We have a specific layer for our machine types, kind of like the meta-crownbay vs meta-menlow layers. Here's my layer config
> LCONF_VERSION = "6"
> BBPATH = "${TOPDIR}"
> BBFILES ?= ""
> BBLAYERS = " \
>    ${HOME}/yocto/poky/meta \
>    ${HOME}/yocto/poky/meta-yocto \
>    ${HOME}/yocto/poky/meta-yocto-bsp \
>    ${HOME}/yocto/poky/meta-intel \
>    ${HOME}/yocto/poky/meta-intel/meta-crownbay \
>    ${HOME}/yocto/poky/meta-oe/meta-oe \
>    ${HOME}/yocto/poky/meta-sonatest \
>    ${HOME}/yocto/poky/meta-sonatest/meta-osprey \
>    "
>
> In meta-osprey, I have a recipes-kernel/linux/linux-yocto_3.4.bbappend which looks like:
> FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
>
> PRINC := "${@int(PRINC) + 9}"
>
> COMPATIBLE_MACHINE_osprey = "osprey"
> KMACHINE_osprey  = "osprey"
> KBRANCH_osprey = "osprey"
>
> SRCREV_machine_pn-linux-yocto_osprey ?= "8eae46d4bf42e2a95694f3774517f2c4aac5af28"
> SRCREV_meta_pn-linux-yocto_osprey ?= "d2afffea88baa8189c7e6adafd74789bf2f41b00"
>
> SRC_URI_osprey = " \
> git://git@gitlab.sonatest.net/yocto/linux-yocto-3-4.git;protocol=ssh;nocheckout=1;branch=${KBRANCH},sapmeta;name=machine,meta \
> file://squashfs.cfg \
> file://hostname.cfg \
> "
>
> LINUX_VERSION = "3.4"
>
> It really looks like what I saw in poky-extra, except for the AUTOREV and the *_pn-linux-yocto_* things.
>
> Couple of questions:
> - What exactly does AUTOREV do?

The fetcher always gets the latest commit on the branches. During
development, we want to pick up everything that is pushed, without
needing to tweak all the SRCREVs.

> - Is it possible to get this bbappend to use the current working directory state (including un-committed changes)
>    using the SRC_URI and AUTOREV?

The SRCREV will only control what the fetcher pulls in, and then
what is checked out. The compile phase will then start. If there
are patches, or other touches to files that aren't committed, they
will be built. But if you have uncommitted changes, you'll have
potential issues with branch switches, etc.

> - what is the _pn_ and why is it absent in the poky-contrib example? (SRCREV_machine_${MACHINE}="${AUTOREV}" and
>    SRCREV_meta="${AUTOREV}")

A bitbake OVERRIDE value. _pn_ "package name". You'll find other
examples floating around the tree, and various layers.

> - in SRC_URI what does the nocheckout=1 and bareclone=[0-1] do? I've had error reports about this in
>    tmp/work/machine/linux-yoctoXYZ/temp logs.

The tools work on the branches, to quickly make a copy of the tree
that has all the branches local, a bareclone and checkout in the
build directory is done. That is what those options control.

>
>> If you apply .cfg fragments in an iherited BSP that disable options
>> in the base BSP (via # CONFIG_FOO is not set), you will be able to
>> turn them off, you of course have to know what the BSP is setting, but
>> if you are deriving work from a board, that's a reasonable thing to do.
>> If you've tied this, and it isn't working, that's a bug.
>
> Yeah I think it works. For the record, I stopped doing this since I find the
> burden of cataloging all things that I need to re-disable isn't worth the
> inheritance. It's best to just know explicitly what I enable or not.

In master, there's another way to do this (it's making it's way into
the docs now)

   include <foo> nocfg

That'll get you the patches, features, etc, but no configuration options.
So you can have the best of both, without needing to turn everything
off.

That being said, we are trying to make the base policy small, sane and
something that we can count on. So if you are finding things that you
think really shouldn't be on .. shout them out, and we can discuss.

>
>> Correct. All of the base branches would already contain those
>> modifications, so you can't trigger then again, unless you are
>> really starting from scratch. If the tools can't determine that
>> a patch is present, it will try to reapply it with the sort of
>> errors that you are seeing if it really does exist.
>
> We did some more experiments and found a lot of confusion around recipe-space's
> "KBRANCH_osprey = x" vs. meta-space's "branch osprey" statements. I have
> observed that if in my gitlab repo, I create a branch called "standard/osprey"
> and this branch is a merge of my-master + emgd-1.16 + standard/crownbay, the
> recipe build tries to patch things like kprobes and such, and basically fails
> saying:
>
> | ERROR: branch standard/osprey was requested, but was not properly
> |        configured to be built. The current branch is standard/osprey/base
>
> 'standard/osprey/base' has not be mentionned anywhere that I can see, it's a
> product of the recipe's work.

It's a property of there being a subbranch of "standard/osprey" being
detected. Due to the way that git stores refs on disk, we have to
rename "standard/osprey" to standard/osprey/base if something like
standard/osprey/bsp is being requested.

>
> If I don't prefix with "standard/" in KBRANCH, then it doesn't patch and things
> seems to work correctly.  (BTW, my gitlab is a clone of your linux-yocto-3.4, we
> just add new branches to that)
>
> Is this behaviour normal? If so, is there documentation for this? Another detail
> when I had the prefix "standard/" in my setup: I have temporarily tried to set
> branch "standard/base" at the same sha1 as my standard/osprey branch, that made
> no difference.

It is normal, and just the tools trying to help you out. The docs
may be slim in this area, I'll double check tomorrow.

>
>>
>> What does your BSP's .scc file look like ? Is it something you
>> can share ?
> bsp/osprey/osprey.scc:
> define KMACHINE osprey
> define KTYPE standard
> define KARCH i386
>
> include ktypes/standard/standard.scc
> branch osprey
>
> kconf hardware bsp/common-sonatest/hardware.cfg
> kconf hardware bsp/common-sonatest/w1.cfg
> kconf hardware osprey.cfg
>
> kconf non-hardware bsp/common-sonatest/non-hardware.cfg
>
> include cfg/x86.scc
> include features/irq/irq.scc
> include features/intel-e1xxxx/intel-e100.scc
> include features/intel-e1xxxx/intel-e1xxxx.scc
> include cfg/dmaengine.scc
> include cfg/8250.scc
> include features/power/intel.scc
>
> include features/usb/ehci-hcd.scc
> include features/usb/ohci-hcd.scc
>
> include features/aufs/aufs.scc
>
> include features/drm-emgd/drm-emgd-1.16.scc
>
> # default policy for standard kernels
> include cfg/usb-mass-storage.scc
> include cfg/boot-live.scc
> include features/latencytop/latencytop.scc
> include features/profiling/profiling.scc
>
> aufs.scc looks like:
> kconf non-hardware aufs.cfg
> branch aufs
>
> and branch aufs exists in my linux clone. It's a merge of upstream-aufs-3.4 +
> v3.4.46 + standard/base. The resulting aufs branch is merged into my machine
> branch in my clone.

aha. And that's what is triggering the problem. Without extra
instructions, branch names are "additive", that's how the hierarchy
gets built up without encoding the names all over the .scc files.

So that "branch aufs" means, create:

   standard/osprey/aufs

.. and hence what I described above must come true.

So you are looking for aufs, just like i have in the 3.8 tree already?
In that case, what you can do is simply create that branch and merge
your changes. i.e. without describing the branch in the .scc file.

If you've already merged it into your BSP branch, no need to merge
it again, but you could just make the aufs.scc be:

   git merge aufs

And your BSP will merge the changes into it's branch on each build.

Cheers,

Bruce

>
>>> Are we going in the wrong direction? I wanted to ask early while I keep digging into this. I have to say I was quite surprised to have difficulty finding similar stories... perhaps I am not using the right keywords in google?
>>>
>>
>> I've got some READMEs on this, but they are not useful by themselves.
>> I'd like to use your use case (which is valid and useful), to clean them
>> up and publish them for the 1.5 release. Perhaps we can even find some
>> gaps in the tools and improve things in the process .. assuming you have
>> the will to work through the case with me!
>
> Absolutely! Tell me what's next...
>
> Cheers!
> /jfd
>