[linux-yocto] Locating kernel metadata

[Bruce Ashfield]

On 13-08-31 12:07 PM, Peter A. Bigot wrote:
> Thanks; I'm starting to understand.  It makes more sense to know this
> all pre-dates Yocto (and maybe its use within bitbake/OE).

heh. It first existed in 2006, in one form or another. The Yocto mapping
is the latest incarnation of the tools. So it has been around for a while
now.

>
> I've generated a lot of text below, but I think the priority for 1.5
> should be to enhance the documention of the current approach, rather
> than try to address any of these issues by changing the tool behavior
> (e.g.  adding overrides).  If the discussion leads to a conclusion that
> the linux-yocto system can be improved then consider such changes for
> early in the 1.6 cycle; in the meantime I'll see what I can do for a
> complex set of recipe-space BSPs within the current system.
>
> Let's start by defining terms that I think don't exist or aren't used
> consistently.  In what follows:

Some of what you may be missing is in the "kernel architecture" manuals,
which I wrote ages ago .. and which we subsequently moved to a harder
to find area of the Yocto site .. since they contained too much detailed
information that wasn't generally useful.

>
> * The YPLKDM is the Yocto Project Linux Kernel Development Manual.
>
> * The "root BSP description" is the "BSP description file that most
> closely matches the KMACHINE and LINUX_KERNEL_TYPE variables passed in
> from the recipe" (YPLKDM).  It is the "top-level kernel feature" that is
> passed to the kernel build subsystem (YPLKDM B.1 step 1).  This should
> be a very special file, as it is this file and anything it recursively
> references, augmented by KERNEL_FEATURES, that defines the final kernel
> configuration.

It can also be augmented by bare .scc/.cfg files on the SRC_URI (aka
recipe space features), which are applied before KERNEL_FEATURES.

>
> * The "kernel metadata root" is the in-tree directory at which searches
> for kernel metadata fragments/patches/descriptions/configuration files
> is rooted. This is ./meta/cfg/kernel-cache as checked out from the
> standard meta branch.

Both definitions are correct, with my minor addition to the first
one.

>
> On 08/30/2013 03:28 PM, Bruce Ashfield wrote:
>> On 13-08-30 03:33 PM, Peter A. Bigot wrote:
>>> There are two search directories documented:
>>>
>>> * "in-tree" which ends up as ${S}/.meta/cfg/kernel-cache through tool
>>> magic guided by KMETA (where the directory name .meta is derived from
>>> the branch identified by ${KMETA})
>>>
>>> * "recipe-space" which ends up as ${WORKDIR} for material identified in
>>> the SRC_URI of a recipe
>>
>> In fact, recipe-space turns into in-tree, since they are all migrated
>> into the tree during the patch/setup phase. Makes things much more
>> uniform and enables easier sharing of features.
>
> Putting everything into one tree before processing is good, but the way
> it's done is not quite consistent with the mental model for the same
> process used in bitbake and recipes.
>
> I'd been thinking of kernel metadata as being managed much the same as
> recipes: It comes from multiple sources (layers), and a developer can
> include, exclude, mask, append, and prioritize to ensure the selected
> contributions are what s/he wants.  Some recipes/extensions get
> submitted up back in open-embedded/poky/meta-oe, and some remain in
> specialized layers without the expectation that other layers may use
> them.  A name identifies a capability, but the user gets to determine
> who supplies its implementation.  Layer-specific modifications are
> expected.

It's not much different than this .. see below.

>
> My current understanding is that kernel metadata is collected from
> multiple sources and combined into a singly-rooted directory hierarchy
> in-tree during patch/setup.  Bare file names will subsequently be
> located from the kernel metadata root within this hierarchy, using a
> to-be-documented search algorithm.  This works because the model assumes
> a single canonical source that holds all metadata so it can be shared,
> coordinated, and managed across all users of the linux-yocto
> infrastructure.  For any given base file name there is only one supplier
> of its capability, with this policy enforced by the maintainer of the
> shared metadata repository (viz., Bruce Ashfield).  Recipe-space
> overrides are, for the most part, an exception.
>
> The difference between these conceptualizations is the fundamental
> disconnect that I think needs to be communicated clearly in the YPLKDM,
> as developers like me who are coming from a linux-mainline or
> linux-gumstix standard recipe approach will expect linux-yocto to be
> substantially the same.

That's a fairly close description of what happens, with the tweak that
the linux-yocto shipped meta data, is an integrated form of an out
of tree "kernel-cache" that is grafted into the tree. This allows the
baseline meta data to follow the tree, and arrive with a single clone
and use model.

Due to the complex interactions of kernel features (and not just
configuration), in order to build, test and consistently combine
features having them all in tree is essential versus being spread out
across multiple disjoint layers and repositories.

But that is only the baseline configuration that ends up in tree,
other layers and recipes are free to extend, modify and mask the
configuration as they see fit. If a feature or fragment is suitable for
more general use, it is encouraged to be merged into the main tree.

Let's put it this way, we can clone the linux-yocto tree, point a directory
full of cross toolchains at it, and then build each and every BSP and
feature combination in a fraction of the time and effort that it would
otherwise take (see the build-all scripts in the kern-tools repo).

>
>>> Earlier documentation assumed that only one or the other is used, but
>>> since recipe-space material can now reference in-tree material a
>>> priority needs to be defined.
>>>
>>> Now: within an scc file you can do:
>>>
>>>     include features/media/media-camera.scc
>>>
>>> and that file in turn can do:
>>>
>>>     kconf media-camera.cfg
>>>
>>> which resolves automatically to features/media/media-camera.cfg. This
>>> reveals that there is also a third search directory: the "current
>>> directory".  Is that the directory in which the fragment that contains
>>> the directive is found?  What is its priority relative to the
>>> recipe-space and in-tree metadata roots?
>>
>> The current directory is searched first, when that fails the rest of
>> the search paths are checked. The order is kernel-cache (in tree)
>> and then the features as they were specified on the SRC_URI.
>
> This seems to be inconsistent with the statement that everything goes
> into the in-tree metadata directory.  After that the SRC_URI should no
> longer be relevant: its prioritization should have applied at the time
> the directory contents were installed.  Preferring in-tree kernel-cache
> content would complicate one of my main use cases, where the in-tree
> kernel-cache has something I don't want and I want to replace it from
> recipe-space rather than attempt to fix what it did after the fact.

Maybe I didn't describe it clearly enough, it really is in that order
that things are searched and used. But what's in the tree is the
baseline data, and what is used to both construct the tree and validate
branches/patches at build time.

After that, it can be extended. In a bitbake format, that means patches,
.cfgs and .scc's specified on the SRC_URI or via KERNEL_FEATURES.

>
> Regardless, developers will need to know exactly how material provided
> by the recipe is prioritized relative to the original "in-tree" material
> and other recipe-space material.  Should the recipe-space material
> provided through SRC_URI in recipe-space follow the same hierarchy as
> cfg/kernel-cache?  Are they copied into the in-tree hierarchy at
> locations determined by the linux-yocto patch/setup tools, or preserving
> the relative hierarchy specified in SRC_URI, or all dumped in as files
> at the top of in-tree hierarchy (as they are normally dumped into
> ${WORKDIR}).

They shouldn't follow the kernel-caches's format, they are free to,
but don't have to. Out of tree .scc, .cfg and patches are kept separate
from in-tree data. They are migrated into the tree structure into
paths that match their recipe space origin and location. That keeps
all the namespaces separate and consistent.

You can't overwrite what's there, or all repeatability and consistency
is gone. Which is the point of the baseline meta data.

>
> Take a patch as an example.  Normally a patch file in a SRC_URI is
> automatically applied to the source tree.  But kernel metadata includes
> patches which (based on the description in section B.1 of the YPLKDM)
> are instead applied only if the patch is identified in a patch directive
> in an scc file that is processed.

Patches are typically not applied from the meta-data, it's there as a
reference for integrated features. Patches are managed via git, and
the .scc files simply validate against the branch that a patch is applied.

>
> This is confusing, and what it suggests to me is that recipe-space
> kernel metadata should not be provided through SRC_URI, because the
> standard semantics of that variable don't match how it needs to be used
> for linux-yocto.  I think it's a bad idea to have files in SRC_URI that
> behave differently for linux-yocto than they would in any other recipe.
> See the proposal outline below for an alternative.

They don't behave any differently than any other recipe .. they are
patched or pushed onto a branch in the order they are specified in
the SRC_URI. What's in the tree is *already* applied.

>
>>> I believe that in the past fragment file names that did not include a
>>> path could be located anywhere under the kernel-cache directory (e.g.,
>>> in cfg, features, patches, ktypes, or even bsp). What if
>>
>> In the distant past, .cfg files had to be in the same directory as
>> their .scc file, the ability to search for them at all is a middle
>> aged feature. To make it easier to write relocatable fragments.
>>
>> That is still the case, but the regex's have been tightened up to
>> prevent false positives. That's why the use of current directory
>> fragments is something that is more and more avoided, but still happens
>> in particular for BSPs when the names of their features tend to be
>> more unique than "foo" :)
>>
>>> media-camera.cfg did not exist in the "current directory" or the root of
>>> any search directory; should it be found if present in a subdirectory?
>>
>> There are existing .scc files that rely on this, so the answer still
>> is 'yes, they should be searched and found'. But again, it is really
>> bad form and trigger false positives, so I understand the concern.
>>
>> I can always update the tools to emit a WARNING when this happens,
>> and then over time restrict the searching. Since good or bad, it
>> is what it is .. for now :)
>>> If so the namespace gets flattened and as more BSPs add their metadata
>>> to the Yocto standard one the likelihood of collision and
>>> incompatibility increases.
>>
>> This can definitely happen. That's why as we collect more meta data
>> fragments, I try and make sure that they are both unique (which is
>> an effort due to fail), have relative paths and are collected in a
>> central location (both for re-use and namespace preservation).
>>
>
> My strongest objection is to the practice of searching within the
> metadata hierarchy to locate a file that doesn't have path components,
> exactly because it requires active management to avoid conflicts.  This
> imposes a burden on people attempting to support a BSP from a layer
> recipe, who in their ignorance may use feature identifiers (pathless
> file names) that may be superseded by the addition of metadata with the
> same base name, possibly in a different path, within the upstream shared
> metadata repository.

That's something that is already being phased out as I mentioned, the
current directory search will absolutely stay, but leaving the current
feature directory to locate a fragment or config won't be possible
in the future.

If you don't look in the current directory (i.e. #include "foo") versus
(#include <foo>), the feature fragments are not relocatable, and that's
a problem with maintenance that is unacceptable.

>
>>>
>>> That last observation identifies a potential fourth directory: The
>>> directory containing the master BSP description file (the one with
>>> KMACHINE/KTYPE/KARCH).  By explicitly calling out this directory any
>>> files in kernel-cache/bsp/* that belong to other BSPs will be ignored,
>>> and media-camera.cfg can be overridden for a specific machine without
>>> having to replicate media-camera.scc.
>>
>> I read this a few times, and I'm not quite sure I'm quite getting it
>> (it's been a long week of 3.10 fun ..)
>>
>> The .scc file that defines a BSP description, is indeed a bit special
>> but the directory that it sits in doesn't have any special meaning after
>> it is located.
>
> If I can convince you to eliminate search for files without paths except
> when in the including file's directory, this won't matter. If I can't,
> then making its directory "special" simplifies the task of a BSP
> collecting all the standard overrides in one location.

See above, that is already happening. I left in my description above,
even after reading this .. just so I can be clear why the current
directory needs to always be on the search path.

>
> In any case, searching the entirety of the bsp subdirectory for features
> seems unstable since if something is re-usable it shouldn't be hidden
> down in a BSP directory, and including those areas increases the chance
> of picking up the wrong implementation of, say, hardware.cfg or wifi.scc.
>
>>> To make kernel metadata deterministic it would be necessary to provide a
>>> mechanism to prioritize these four search directories, and perhaps also
>>
>> I'm all for determinism. As a start, I can get the current algorithm
>> into the docs .. honestly, I'm happy that someone cares to talk about
>> it, no one has asked in the past, so it's not documented at the moment.
>>
>>> to change the algorithm so that unqualified files (no path) will not be
>>> found in subdirectories of these directories.
>>
>> Aha. I think I know what you meant above about the BSP directories.
>> You'd want them to be excluded, and the current BSP's could override
>> a generic one. If the .cfg is added with a relative path, that of
>> course wouldn't work, but if it is one that is searched, there's a
>> possibility of doing something like that.
>>
>> I'm hesitant to give them that much of a special status .. but I'll
>> think on it over the weekend.
>>
>> My first thought is to just restore some much older functionality
>> (i.e. pre-yocto .. who knew there was such a long, twisted story?)
>> that offered an "override" syntax.
>>
>> We have:
>>
>>   force kconf non-hardware foo.cfg
>>
>> Which ensure that an option makes it into any includes, and with
>> override, you also have:
>>
>>   override kconf non-hardware foo.cfg myfoo.cfg
>>
>> Where the .cfg's can be specified by their relative path to the root
>> of the search dir.
>>
>> As it stands, you can obviously let the common one be included and
>> then override the options that you don't want / like in your board
>> configuration.
>
> As I mentioned I don't think changes are appropriate at this stage of
> the 1.5 development cycle, and I'd like to see how far I can get with
> the "let common do what it wants and try to clean up later" approach,
> though philosophically I dislike it since in the future "common" may do
> something I didn't anticipate in my cleanup.

The change would be completely transparent to any existing meta
data and users, so there's no risk. I may still bring it in, since the
code already exists :)

>
> However, it may be possible to change things in 1.6 to make kernel
> metadata management more consistent with recipe searches while not
> disrupting existing practice too much.  Here's a rough outline of a
> candidate process:
>
> SRC_URI is used exactly the same as in other recipes.  It normally
> contains a single entry which is the location of the git repository,
> with KBRANCH and maybe KMETA branches or directories.  If it does have
> other stuff (such as patches), it is treated as in other recipes.
>
> KMETA behaves exactly as it does now, if that means it ensures there is
> a directory of metadata placed somewhere in-tree.  I expect the root of
> that directory to contain the cfg, patches, and scripts directories just
> as the root of the meta directory in the meta branch of yocto-3.10 does,
> and that within it is the kernel metadata root (cfg/kernel-cache).
>
> SRC_KMETA_URI is a special variable which contains the recipe-space
> metadata currently supplied through SRC_URI.  Its contents are
> transferred into the kernel metadata root instead of ${WORKDIR}. The

Everything bounces through ${WORKDIR}, that's how the fetcher works,
and we want the fetcher to take care of the details of getting SRC_URI
entries and making them available. It works well, and there's no reason
to duplicate its functionality.

It's after they are in WORKDIR they make it into the kernel meta
directory (in their own namespace).

But that detail is largely irrelevant.

I already have a feature that slipped from 1.5 that would allow full
recipe space directory hierarchies to be specified, without any extra
variables or special processing. Simply put the directory on the
SRC_URI and the processing triggers, it'll go in early in the 1.6 cycle.
I have TomZ's email asking me about it from 2012 staring at me!

> URIs are not applied as with SRC_URI even if they are patches, but tar
> files, git branches, and other archives should be unpacked over the
> previous contents preserving any internal hierarchy (support

This is all fetcher functionality already, not something we want to
duplicate.

> striplevel).  Files should be copied by shifting so that if
> SRC_KMETA_URI identifies cfg/kernel-cache/features/media/v4l2.cfg as an
> input the corresponding v4l2.cfg will be found at features/meta/v4l2.cfg

I read this about 4 times, and it still doesn't read properly to me.
So I'm not sure what you are trying to describe here.

> by the description parsing code.  All overriding will be done in the
> order of appearance in SRC_KMETA_URI.

If I'm reading it right, it can't happen as you've described it, plus
special kernel specific variables are bad. I've eliminated pretty much
every one that I had to introduce in yocto 0.9 to now and new ones are
a hard sell.

What happens in recipe space and extensions cannot directly clobber
what's already in the tree. Validation will fail, patch failures will
occur and many other types of issues .. I've lived through them all, and
won't go back. They must be explicitly called out in .scc files, where
the writer of the file knows what they are doing.

>
> This produces a single in-tree metadata directory, without attempting to
> interpret the contents of any of the files.  It should be a separate
> task in the recipe, allowing the user to manipulate the recipe until the
> contents of that metadata hierarchy are as desired.

The user shouldn't be interpreting the hierarchy or manipulating it at
all .. just overriding from their own .scc files. Anything else is
complicated and binds everything to a very specific structure.

>
> The root BSP description is then identified per YPLKDM section B.1,
> possibly restricted to search the bsp subhierarchy of the kernel
> metadata root.  It is processed, followed by processing the implicit
> include of everything in KERNEL_FEATURES.
>
> In all cases where a directive has an unadorned file like:
>
>    include foo

This part has already been happening for sometime, and will more than
likely be the case regardless in 1.6.

>
> the tools must find file foo in the directory containing the file from
> which the directive was read (the kernel metadata root for files in
> KERNEL_FEATURES).  Failure to find the file in that directory is an
> error.  Override is possible because the recipe could ensure the desired
> version of that file was written into place through SRC_KMETA_URI.

If you mean the overwriting of "foo" by the outside override, no, that
won't happen.

>
> If the directive specifies the file with a path, like:
>
>    kconf features/media/v4l2.cfg
>
> then it must be found following that path from the kernel metadata
> root.  Again override through SRC_KMETA_URI works.
>
> I understand this would be a change requiring BSPs to update their
> configuration files.  I also don't see that task taking more than five

I can make the searching changes in 1.6, as they were already planned,
but anything more invasive than that is a problem, since I don't have
access to every BSP description that exists :)

The original design of all this, is that any feature, any BSP can
manipulate was has already been specified in the baseline configuration
.. to a point. You can't mess with patches, git merges or anything that
modifies the structure of the baseline tree. Horrible patch errors ensue 
and the
entire thing collapses under it's own weight. The current system was
created after watching patch and configuration management collapse.

What *any* feature can do, is influence the configuration and ordering
of configs and optional features. That is done via commands and
operations in their .scc files. The entire base configuration can be in
a read only location, and you can still manipulate it.

That's what the 'override' command that I mentioned does, as do some
other "trigger" or mapping commands.

A .scc file injected from an external location (i.e recipe space) can do
this:

   override features/media/v4l2.cfg my_features/media/v4l2.cfg

or

   override features/edf/patch01.patch my_features/edf/abetterpatch.foo

.. etc.

The other part of the entire model is that if the baseline configuration,
features, patches, etc are NOT useful to someone's BSP or feature, then
they could be wrong, we should discuss and adjust the descriptions
accordingly. i.e. the need to override and manipulate goes away over
time, with the override mechanisms filling the gaps temporarily.

Great discussion BTW, I really do appreciate it, kept me up until 1:30 
AM .. again!

Cheers,

Bruce


> or ten minutes, so it should be worth the effort to produce a more
> consistent model of how the processing is done.
>
> Peter
>
>> Cheers,
>>
>> Bruce
>>
>>>
>>> Comments and discussion?
>>>
>>> Peter
>>>
>>> [1]
>>> http://article.gmane.org/gmane.comp.handhelds.openembedded.core/41298/
>>> [2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=5090
>>> _______________________________________________
>>> linux-yocto mailing list
>>> linux-yocto at yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/linux-yocto
>>
>