[linux-yocto] Locating kernel metadata

[Peter A. Bigot]

Thanks; I'm starting to understand.  It makes more sense to know this 
all pre-dates Yocto (and maybe its use within bitbake/OE).

I've generated a lot of text below, but I think the priority for 1.5 
should be to enhance the documention of the current approach, rather 
than try to address any of these issues by changing the tool behavior 
(e.g.  adding overrides).  If the discussion leads to a conclusion that 
the linux-yocto system can be improved then consider such changes for 
early in the 1.6 cycle; in the meantime I'll see what I can do for a 
complex set of recipe-space BSPs within the current system.

Let's start by defining terms that I think don't exist or aren't used 
consistently.  In what follows:

* The YPLKDM is the Yocto Project Linux Kernel Development Manual.

* The "root BSP description" is the "BSP description file that most 
closely matches the KMACHINE and LINUX_KERNEL_TYPE variables passed in 
from the recipe" (YPLKDM).  It is the "top-level kernel feature" that is 
passed to the kernel build subsystem (YPLKDM B.1 step 1).  This should 
be a very special file, as it is this file and anything it recursively 
references, augmented by KERNEL_FEATURES, that defines the final kernel 
configuration.

* The "kernel metadata root" is the in-tree directory at which searches 
for kernel metadata fragments/patches/descriptions/configuration files 
is rooted. This is ./meta/cfg/kernel-cache as checked out from the 
standard meta branch.

On 08/30/2013 03:28 PM, Bruce Ashfield wrote:
> On 13-08-30 03:33 PM, Peter A. Bigot wrote:
>> There are two search directories documented:
>>
>> * "in-tree" which ends up as ${S}/.meta/cfg/kernel-cache through tool
>> magic guided by KMETA (where the directory name .meta is derived from
>> the branch identified by ${KMETA})
>>
>> * "recipe-space" which ends up as ${WORKDIR} for material identified in
>> the SRC_URI of a recipe
>
> In fact, recipe-space turns into in-tree, since they are all migrated
> into the tree during the patch/setup phase. Makes things much more
> uniform and enables easier sharing of features.

Putting everything into one tree before processing is good, but the way 
it's done is not quite consistent with the mental model for the same 
process used in bitbake and recipes.

I'd been thinking of kernel metadata as being managed much the same as 
recipes: It comes from multiple sources (layers), and a developer can 
include, exclude, mask, append, and prioritize to ensure the selected 
contributions are what s/he wants.  Some recipes/extensions get 
submitted up back in open-embedded/poky/meta-oe, and some remain in 
specialized layers without the expectation that other layers may use 
them.  A name identifies a capability, but the user gets to determine 
who supplies its implementation.  Layer-specific modifications are expected.

My current understanding is that kernel metadata is collected from 
multiple sources and combined into a singly-rooted directory hierarchy 
in-tree during patch/setup.  Bare file names will subsequently be 
located from the kernel metadata root within this hierarchy, using a 
to-be-documented search algorithm.  This works because the model assumes 
a single canonical source that holds all metadata so it can be shared, 
coordinated, and managed across all users of the linux-yocto 
infrastructure.  For any given base file name there is only one supplier 
of its capability, with this policy enforced by the maintainer of the 
shared metadata repository (viz., Bruce Ashfield).  Recipe-space 
overrides are, for the most part, an exception.

The difference between these conceptualizations is the fundamental 
disconnect that I think needs to be communicated clearly in the YPLKDM, 
as developers like me who are coming from a linux-mainline or 
linux-gumstix standard recipe approach will expect linux-yocto to be 
substantially the same.

>> Earlier documentation assumed that only one or the other is used, but
>> since recipe-space material can now reference in-tree material a
>> priority needs to be defined.
>>
>> Now: within an scc file you can do:
>>
>>     include features/media/media-camera.scc
>>
>> and that file in turn can do:
>>
>>     kconf media-camera.cfg
>>
>> which resolves automatically to features/media/media-camera.cfg. This
>> reveals that there is also a third search directory: the "current
>> directory".  Is that the directory in which the fragment that contains
>> the directive is found?  What is its priority relative to the
>> recipe-space and in-tree metadata roots?
>
> The current directory is searched first, when that fails the rest of
> the search paths are checked. The order is kernel-cache (in tree)
> and then the features as they were specified on the SRC_URI.

This seems to be inconsistent with the statement that everything goes 
into the in-tree metadata directory.  After that the SRC_URI should no 
longer be relevant: its prioritization should have applied at the time 
the directory contents were installed.  Preferring in-tree kernel-cache 
content would complicate one of my main use cases, where the in-tree 
kernel-cache has something I don't want and I want to replace it from 
recipe-space rather than attempt to fix what it did after the fact.

Regardless, developers will need to know exactly how material provided 
by the recipe is prioritized relative to the original "in-tree" material 
and other recipe-space material.  Should the recipe-space material 
provided through SRC_URI in recipe-space follow the same hierarchy as 
cfg/kernel-cache?  Are they copied into the in-tree hierarchy at 
locations determined by the linux-yocto patch/setup tools, or preserving 
the relative hierarchy specified in SRC_URI, or all dumped in as files 
at the top of in-tree hierarchy (as they are normally dumped into 
${WORKDIR}).

Take a patch as an example.  Normally a patch file in a SRC_URI is 
automatically applied to the source tree.  But kernel metadata includes 
patches which (based on the description in section B.1 of the YPLKDM) 
are instead applied only if the patch is identified in a patch directive 
in an scc file that is processed.

This is confusing, and what it suggests to me is that recipe-space 
kernel metadata should not be provided through SRC_URI, because the 
standard semantics of that variable don't match how it needs to be used 
for linux-yocto.  I think it's a bad idea to have files in SRC_URI that 
behave differently for linux-yocto than they would in any other recipe.  
See the proposal outline below for an alternative.

>> I believe that in the past fragment file names that did not include a
>> path could be located anywhere under the kernel-cache directory (e.g.,
>> in cfg, features, patches, ktypes, or even bsp). What if
>
> In the distant past, .cfg files had to be in the same directory as
> their .scc file, the ability to search for them at all is a middle
> aged feature. To make it easier to write relocatable fragments.
>
> That is still the case, but the regex's have been tightened up to
> prevent false positives. That's why the use of current directory
> fragments is something that is more and more avoided, but still happens
> in particular for BSPs when the names of their features tend to be
> more unique than "foo" :)
>
>> media-camera.cfg did not exist in the "current directory" or the root of
>> any search directory; should it be found if present in a subdirectory?
>
> There are existing .scc files that rely on this, so the answer still
> is 'yes, they should be searched and found'. But again, it is really
> bad form and trigger false positives, so I understand the concern.
>
> I can always update the tools to emit a WARNING when this happens,
> and then over time restrict the searching. Since good or bad, it
> is what it is .. for now :)
>> If so the namespace gets flattened and as more BSPs add their metadata
>> to the Yocto standard one the likelihood of collision and
>> incompatibility increases.
>
> This can definitely happen. That's why as we collect more meta data
> fragments, I try and make sure that they are both unique (which is
> an effort due to fail), have relative paths and are collected in a
> central location (both for re-use and namespace preservation).
>

My strongest objection is to the practice of searching within the 
metadata hierarchy to locate a file that doesn't have path components, 
exactly because it requires active management to avoid conflicts.  This 
imposes a burden on people attempting to support a BSP from a layer 
recipe, who in their ignorance may use feature identifiers (pathless 
file names) that may be superseded by the addition of metadata with the 
same base name, possibly in a different path, within the upstream shared 
metadata repository.

>>
>> That last observation identifies a potential fourth directory: The
>> directory containing the master BSP description file (the one with
>> KMACHINE/KTYPE/KARCH).  By explicitly calling out this directory any
>> files in kernel-cache/bsp/* that belong to other BSPs will be ignored,
>> and media-camera.cfg can be overridden for a specific machine without
>> having to replicate media-camera.scc.
>
> I read this a few times, and I'm not quite sure I'm quite getting it
> (it's been a long week of 3.10 fun ..)
>
> The .scc file that defines a BSP description, is indeed a bit special
> but the directory that it sits in doesn't have any special meaning after
> it is located.

If I can convince you to eliminate search for files without paths except 
when in the including file's directory, this won't matter. If I can't, 
then making its directory "special" simplifies the task of a BSP 
collecting all the standard overrides in one location.

In any case, searching the entirety of the bsp subdirectory for features 
seems unstable since if something is re-usable it shouldn't be hidden 
down in a BSP directory, and including those areas increases the chance 
of picking up the wrong implementation of, say, hardware.cfg or wifi.scc.

>> To make kernel metadata deterministic it would be necessary to provide a
>> mechanism to prioritize these four search directories, and perhaps also
>
> I'm all for determinism. As a start, I can get the current algorithm 
> into the docs .. honestly, I'm happy that someone cares to talk about
> it, no one has asked in the past, so it's not documented at the moment.
>
>> to change the algorithm so that unqualified files (no path) will not be
>> found in subdirectories of these directories.
>
> Aha. I think I know what you meant above about the BSP directories.
> You'd want them to be excluded, and the current BSP's could override
> a generic one. If the .cfg is added with a relative path, that of
> course wouldn't work, but if it is one that is searched, there's a
> possibility of doing something like that.
>
> I'm hesitant to give them that much of a special status .. but I'll
> think on it over the weekend.
>
> My first thought is to just restore some much older functionality
> (i.e. pre-yocto .. who knew there was such a long, twisted story?)
> that offered an "override" syntax.
>
> We have:
>
>   force kconf non-hardware foo.cfg
>
> Which ensure that an option makes it into any includes, and with
> override, you also have:
>
>   override kconf non-hardware foo.cfg myfoo.cfg
>
> Where the .cfg's can be specified by their relative path to the root
> of the search dir.
>
> As it stands, you can obviously let the common one be included and
> then override the options that you don't want / like in your board
> configuration.

As I mentioned I don't think changes are appropriate at this stage of 
the 1.5 development cycle, and I'd like to see how far I can get with 
the "let common do what it wants and try to clean up later" approach, 
though philosophically I dislike it since in the future "common" may do 
something I didn't anticipate in my cleanup.

However, it may be possible to change things in 1.6 to make kernel 
metadata management more consistent with recipe searches while not 
disrupting existing practice too much.  Here's a rough outline of a 
candidate process:

SRC_URI is used exactly the same as in other recipes.  It normally 
contains a single entry which is the location of the git repository, 
with KBRANCH and maybe KMETA branches or directories.  If it does have 
other stuff (such as patches), it is treated as in other recipes.

KMETA behaves exactly as it does now, if that means it ensures there is 
a directory of metadata placed somewhere in-tree.  I expect the root of 
that directory to contain the cfg, patches, and scripts directories just 
as the root of the meta directory in the meta branch of yocto-3.10 does, 
and that within it is the kernel metadata root (cfg/kernel-cache).

SRC_KMETA_URI is a special variable which contains the recipe-space 
metadata currently supplied through SRC_URI.  Its contents are 
transferred into the kernel metadata root instead of ${WORKDIR}. The 
URIs are not applied as with SRC_URI even if they are patches, but tar 
files, git branches, and other archives should be unpacked over the 
previous contents preserving any internal hierarchy (support 
striplevel).  Files should be copied by shifting so that if 
SRC_KMETA_URI identifies cfg/kernel-cache/features/media/v4l2.cfg as an 
input the corresponding v4l2.cfg will be found at features/meta/v4l2.cfg 
by the description parsing code.  All overriding will be done in the 
order of appearance in SRC_KMETA_URI.

This produces a single in-tree metadata directory, without attempting to 
interpret the contents of any of the files.  It should be a separate 
task in the recipe, allowing the user to manipulate the recipe until the 
contents of that metadata hierarchy are as desired.

The root BSP description is then identified per YPLKDM section B.1, 
possibly restricted to search the bsp subhierarchy of the kernel 
metadata root.  It is processed, followed by processing the implicit 
include of everything in KERNEL_FEATURES.

In all cases where a directive has an unadorned file like:

   include foo

the tools must find file foo in the directory containing the file from 
which the directive was read (the kernel metadata root for files in 
KERNEL_FEATURES).  Failure to find the file in that directory is an 
error.  Override is possible because the recipe could ensure the desired 
version of that file was written into place through SRC_KMETA_URI.

If the directive specifies the file with a path, like:

   kconf features/media/v4l2.cfg

then it must be found following that path from the kernel metadata 
root.  Again override through SRC_KMETA_URI works.

I understand this would be a change requiring BSPs to update their 
configuration files.  I also don't see that task taking more than five 
or ten minutes, so it should be worth the effort to produce a more 
consistent model of how the processing is done.

Peter

> Cheers,
>
> Bruce
>
>>
>> Comments and discussion?
>>
>> Peter
>>
>> [1] 
>> http://article.gmane.org/gmane.comp.handhelds.openembedded.core/41298/
>> [2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=5090
>> _______________________________________________
>> linux-yocto mailing list
>> linux-yocto at yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/linux-yocto
>