[yocto] dynamic layer dependencies in meta-security

[Patrick Ohly]

Hello Armin!

I'm trying to use the latest meta-security master in refkit. Refkit runs
yocto-compat-layer.py against its own layers, which then pulls in
meta-security as one of the dependencies. That's where I am hitting a
snag:

INFO: Detected layers:
INFO: meta-refkit: LayerType.DISTRO, /fast/work/intel-iot-refkit-pr/meta-refkit
INFO: 
INFO: Setting up for meta-refkit(LayerType.DISTRO), /fast/work/intel-iot-refkit-pr/meta-refkit
INFO: Adding layer dependency meta-selftest
INFO: Adding layer dependency meta-intel
INFO: Adding layer dependency meta-refkit-gateway
INFO: Adding layer dependency meta-refkit-core
INFO: Adding layer dependency meta-oic
INFO: Adding layer dependency meta-iot-web
INFO: Adding layer dependency meta-refkit-computervision
INFO: Adding layer dependency meta-intel-realsense
INFO: Adding layer dependency meta-clang
INFO: Adding layer dependency meta-oe
INFO: Adding layer dependency meta-refkit-industrial
INFO: Adding layer dependency meta-python
INFO: Adding layer dependency meta-ros
INFO: Adding layer dependency meta-iotqa
INFO: Adding layer dependency meta-security
INFO: Adding layer dependency meta-perl
INFO: Adding layer dependency meta-security-isafw
INFO: Adding layer dependency meta-filesystems
INFO: Adding layer dependency meta-gnome
INFO: Adding layer dependency meta-networking
INFO: Adding layer dependency meta-flatpak
Traceback (most recent call last):
  File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 203, in <module>
    ret =  main()
  File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 151, in main
    if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger) or \
  File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 195, in add_layer_dependencies
    output = check_command('Getting existing layers failed.', 'bitbake-layers show-layers').decode('utf-8')
  File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 228, in check_command
    raise RuntimeError(msg)
RuntimeError: Getting existing layers failed.
Command: bitbake-layers show-layers
Output:
ERROR: Layer 'security' depends on layer 'xfce-layer', but this layer is not enabled in your configuration

The reason is the way how meta-security declares its dependencies:

LAYERDEPENDS_security = "core openembedded-layer perl-layer"
LAYERDEPENDS_security += "${@bb.utils.contains("DISTRO_FEATURES", "x11", "gnome-layer xfce-layer", "", d)}"
LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm", "tpm-layer", "",d)}"
LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm2", "tpm-layer", "",d)}"
LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "vtpm", "tpm-layer meta-filesystems", "",d)}"

I haven't checked how exactly yocto-compat-layer.py deals with this, but
the end result is that it doesn't add meta-xfce when adding
meta-security. Then when "bitbake-layers show-layers" is called, a
distro (poky) has been chosen where x11 is in DISTRO_FEATURES, causing
the dependency check to fail.

This highlights a conceptual issue with the dynamic
LAYERDEPENDS_security above: which distro or machine config is active?
Tools like yocto-compat-layer.py or the layer index have to do their
work before that is defined.

This is also problematic with multiconfig. The same bblayers.conf might
be used with one config where x11 is off and another where it is on.

I suspect LAYERDEPENDS simply can't be dynamic. It either has to include
everything that a layer might depend on (even when the additional
content doesn't end up being used), or it just depends on the essential
parts and then enables additional recipes only when the optional layers
they depend on are present. BBFILES_DYNAMIC can be used for that, or the
recipes themselves check BBCOLLECTIONS in anonymous python code and
raise a SkipRecipe exception when their pre-condition isn't satisfied.

Copying Paul (layer index) and Mark (oe-yocto-compat.py, setup tool) for
their input.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.