[yocto] Layer Priority with Wildcard .bbappend Files

[Paul Eggleton]

Hi Bob / Nick,

On Monday 24 November 2014 16:25:58 Bob Cochran wrote:
> On 11/24/2014 03:22 PM, Stevens, Nick wrote:
> > I think I've encountered a bug with how multiple bbappend files are
> > processed when one of the bbappends contains a filename wildcard, but I
> > want to make sure there's not something I'm missing before filing a bug
> > report.> 
> > I have a BSP layer and a customization layer that are based on the OE/poky
> > Daisy release. The output of `bitbake-layers show-layers` looks like:
> >      layer                 path                 priority
> >      =====================================================
> >      meta                  poky/meta            5
> >      meta-yocto            poky/meta-yocto      5
> >      meta-yocto-bsp        poky/meta-yocto-bsp  5
> >      ...ellided...
> >      meta-bsp              meta-bsp             6
> >      meta-custom           meta-custom          8
> > 
> > In meta-bsp there is a bbappend for base-files named base-
> > files_3.0.14.bbappend. The meta-bsp bbappend adds a sysctl.conf to /etc -
> > pretty straightforward:
> >      FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
> >      SRC_URI += "file://sysctl.conf"
> >      do_install_append() {
> >      
> >          install -m 0644 ${WORKDIR}/sysctl.conf ${D}${sysconfdir}/
> >      
> >      }
> > 
> > Now what I want to do is add a file called base-files_%.bbappend to meta-
> > custom. It has its own version of sysctl.conf, and the recipe looks like
> > this:
> >      FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
> > 
> > Here's where things get weird. If I run `bitbake -e base-files` and pull
> > out the section for FILESEXTRAPATHS, this is what I get (note that I've
> > stripped out the huge absolute paths to make this easier to read):
> >      # $FILESEXTRAPATHS [5 operations]
> >      #   set poky/meta/conf/documentation.conf:172
> >      #     [doc] "Extends the search path the OpenEmbedded build system
> >      uses when looking for files and patches as it processes recipes and
> >      append files." #   _prepend
> >      meta-custom/recipes-core/base-files/base-files_%.bbappend:6 #    
> >      "meta-custom/recipes-core/base-files/base-files:"
> >      #   _prepend
> >      meta-bsp/recipes-core/base-files/base-files_3.0.14.bbappend:3
> >      #     "meta-bsp/recipes-core/base-files/base-files:"
> >      #   set data_smart.py:432 [finalize]
> >      #     "meta-custom/recipes-core/base-files/base-files:"
> >      #   set data_smart.py:432 [finalize]
> >      #    
> >      "meta-bsp/recipes-core/base-files/base-files:meta-custom/recipes-cor
> >      e/base-files/base-files:" # computed:
> >      #  
> >      "meta-bsp/recipes-core/base-files/base-files:meta-custom/recipes-cor
> >      e/base-files/base-files:"
> >      FILESEXTRAPATHS="meta-bsp/recipes-core/base-files/base-files:meta-cu
> >      stom/recipes-core/base-files/base-files:"> 
> > For some reason meta-bsp is coming before meta-custom in FILESEXTRAPATHS,
> > even though meta-custom has a higher priority!
>
> I also have some questions about how FILESEXTRAPATHS is supposed to work
> with append files and layer priorities.
> 
> Does a higher layer priority mean that the FILESEXTRAPATHS operation
> should occur first?  

No. bbappends are parsed in ascending layer priority order. It wouldn't make 
sense to do it in the reverse - you want the higher priority layer's settings 
to take precedence

> If this is the case, then a lower priority layer prepend will appear to the
> left of the higher layer prepend since it runs later, and this will probably
> not provide the result you want.

The highest priority layer's bbappend will prepend last, thus anything it 
prepends will be first in the list.

> The prepend and layer logic seems messy to me.  I would like to have a
> way to write my custom layer and specify that what I include in my
> SRC_URI in each recipe is definitive and can not be overwritten.
> Suggestions on how to best accomplish this will be greatly appreciated.

I think this is pretty much already the case. Is the problem that you can't 
follow the current behaviour, or that you don't completely trust the layers 
you are pulling in?
 
> I have been trying to accomplish this with FILESOVERRIDES, but this
> logic seems counterintuitive since  DISTROOVERRIDES take precedence over
> MACHINEOVERRIDES in building the file search path during the unpack task.
> 
> It seems to me that the file search path should be built using
> FILESOVERRIDES from left to right:  TRANSLATED_TARGET_ARCH,
> MACHINEOVERRIDES, and then DISTROOVERRIDES (specific to generic), but
> this isn't what I'm seeing.  Still investigating how it's all put
> together...
> 
> 
> I verified this by building an image - the sysctl.conf that ends up in
> the final image is the one from meta-bsp, not the one from meta-custom.
> But if I switch the name of base-files_%.bbappend in meta-custom to
> base-files_3.0.14.bbappend, this is what I get:
> >      # $FILESEXTRAPATHS [5 operations]
> >      #   set poky/meta/conf/documentation.conf:172
> >      #     [doc] "Extends the search path the OpenEmbedded build system
> >      uses when looking for files and patches as it processes recipes and
> >      append files." #   _prepend
> >      meta-bsp/recipes-core/base-files/base-files_3.0.14.bbappend:3 #    
> >      "meta-bsp/recipes-core/base-files/base-files:"
> >      #   _prepend
> >      meta-custom/recipes-core/base-files/base-files_3.0.14.bbappend:6 #  
> >        "meta-custom/recipes-core/base-files/base-files:"
> >      #   set data_smart.py:432 [finalize]
> >      #     "meta-bsp/recipes-core/base-files/base-files:"
> >      #   set data_smart.py:432 [finalize]
> >      #    
> >      "meta-custom/recipes-core/base-files/base-files:meta-bsp/recipes-cor
> >      e/base-files/base-files:" # computed:
> >      #  
> >      "meta-custom/recipes-core/base-files/base-files:meta-bsp/recipes-cor
> >      e/base-files/base-files:"
> >      FILESEXTRAPATHS="meta-custom/recipes-core/base-files/base-files:meta
> >      -bsp/recipes-core/base-files/base-files:"> 
> > And now the sysctl.conf file is being pulled from meta-custom.

That absolutely should not happen. If you change it back does it revert to the 
version from meta-bsp? The wildcarding should not change the order in which 
the files are applied. The only corner case I can think of would be if the 
layer priority values are the same number - this isn't the case is it?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre