[linux-yocto] Locating kernel metadata

[Bruce Ashfield]

On 13-08-30 03:33 PM, Peter A. Bigot wrote:
> I want the ability to have a BSP layer linux-yocto_3.10.bbappend which
> supports a new machine by providing a BSP scc file and some BSP-specific
> overrides in recipe space that re-use and where necessary override
> features that exist in the standard meta hierarchy.  There is some
> previous discussion of this at [1] and [2].  What remains unclear to me
> are two issues:
>
> 1) how to specify and prioritize the search directories for metadata
> material (patches, fragments, features, kernel types);
>
> 2) how to identify the file containing the material within a directive
> (viz., with or without a path relative to a root search directory);
>
> There are two search directories documented:
>
> * "in-tree" which ends up as ${S}/.meta/cfg/kernel-cache through tool
> magic guided by KMETA (where the directory name .meta is derived from
> the branch identified by ${KMETA})
>
> * "recipe-space" which ends up as ${WORKDIR} for material identified in
> the SRC_URI of a recipe

In fact, recipe-space turns into in-tree, since they are all migrated
into the tree during the patch/setup phase. Makes things much more
uniform and enables easier sharing of features.

>
> Earlier documentation assumed that only one or the other is used, but
> since recipe-space material can now reference in-tree material a
> priority needs to be defined.
>
> Now: within an scc file you can do:
>
>     include features/media/media-camera.scc
>
> and that file in turn can do:
>
>     kconf media-camera.cfg
>
> which resolves automatically to features/media/media-camera.cfg. This
> reveals that there is also a third search directory: the "current
> directory".  Is that the directory in which the fragment that contains
> the directive is found?  What is its priority relative to the
> recipe-space and in-tree metadata roots?

The current directory is searched first, when that fails the rest of
the search paths are checked. The order is kernel-cache (in tree)
and then the features as they were specified on the SRC_URI.

>
> I believe that in the past fragment file names that did not include a
> path could be located anywhere under the kernel-cache directory (e.g.,
> in cfg, features, patches, ktypes, or even bsp). What if

In the distant past, .cfg files had to be in the same directory as
their .scc file, the ability to search for them at all is a middle
aged feature. To make it easier to write relocatable fragments.

That is still the case, but the regex's have been tightened up to
prevent false positives. That's why the use of current directory
fragments is something that is more and more avoided, but still happens
in particular for BSPs when the names of their features tend to be
more unique than "foo" :)

> media-camera.cfg did not exist in the "current directory" or the root of
> any search directory; should it be found if present in a subdirectory?

There are existing .scc files that rely on this, so the answer still
is 'yes, they should be searched and found'. But again, it is really
bad form and trigger false positives, so I understand the concern.

I can always update the tools to emit a WARNING when this happens,
and then over time restrict the searching. Since good or bad, it
is what it is .. for now :)

> If so the namespace gets flattened and as more BSPs add their metadata
> to the Yocto standard one the likelihood of collision and
> incompatibility increases.

This can definitely happen. That's why as we collect more meta data
fragments, I try and make sure that they are both unique (which is
an effort due to fail), have relative paths and are collected in a
central location (both for re-use and namespace preservation).

>
> That last observation identifies a potential fourth directory: The
> directory containing the master BSP description file (the one with
> KMACHINE/KTYPE/KARCH).  By explicitly calling out this directory any
> files in kernel-cache/bsp/* that belong to other BSPs will be ignored,
> and media-camera.cfg can be overridden for a specific machine without
> having to replicate media-camera.scc.

I read this a few times, and I'm not quite sure I'm quite getting it
(it's been a long week of 3.10 fun ..)

The .scc file that defines a BSP description, is indeed a bit special
but the directory that it sits in doesn't have any special meaning after
it is located.

>
> To make kernel metadata deterministic it would be necessary to provide a
> mechanism to prioritize these four search directories, and perhaps also

I'm all for determinism. As a start, I can get the current algorithm 
into the docs .. honestly, I'm happy that someone cares to talk about
it, no one has asked in the past, so it's not documented at the moment.

> to change the algorithm so that unqualified files (no path) will not be
> found in subdirectories of these directories.

Aha. I think I know what you meant above about the BSP directories.
You'd want them to be excluded, and the current BSP's could override
a generic one. If the .cfg is added with a relative path, that of
course wouldn't work, but if it is one that is searched, there's a
possibility of doing something like that.

I'm hesitant to give them that much of a special status .. but I'll
think on it over the weekend.

My first thought is to just restore some much older functionality
(i.e. pre-yocto .. who knew there was such a long, twisted story?)
that offered an "override" syntax.

We have:

   force kconf non-hardware foo.cfg

Which ensure that an option makes it into any includes, and with
override, you also have:

   override kconf non-hardware foo.cfg myfoo.cfg

Where the .cfg's can be specified by their relative path to the root
of the search dir.

As it stands, you can obviously let the common one be included and
then override the options that you don't want / like in your board
configuration.

Cheers,

Bruce

>
> Comments and discussion?
>
> Peter
>
> [1] http://article.gmane.org/gmane.comp.handhelds.openembedded.core/41298/
> [2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=5090
> _______________________________________________
> linux-yocto mailing list
> linux-yocto at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/linux-yocto