[yocto] [meta-selinux][PATCH] conf/layer.conf: use BBFILES_DYNAMIC for dynamic layers

[Yi Zhao]

On 9/10/19 1:11 AM, Joe MacDonald wrote:
> Hi Yi,
>
> [[meta-selinux][PATCH] conf/layer.conf: use BBFILES_DYNAMIC for dynamic layers] On 19.09.09 (Mon 14:01) Yi Zhao wrote:
>
>> From: Robert Yang <liezhi.yang at windriver.com>
>>
>> The previous code add all BBFILE_COLLECTIONS/recipes*/*/*.bbappend to BBFILES,
>> which causes the parsing very slow when there are many layers, e.g., I have 87
>> layers:
>>
>> * Before:
>> $ rm -fr tmp-glibc/ cache; time bitbake -p
>> real    0m45.173s
>> user    0m0.560s
>> sys     0m0.060s
>>
>> * After:
>> $ rm -fr tmp-glibc/ cache; time bitbake -p
>> real    0m25.542s
>> user    0m0.572s
>> sys     0m0.040s
>>
>> It wasted 20s which wasn't worth (The host has 128 threads, it should cost more
>> time on less power host), use BBFILES_DYNAMIC can fix the problem.
>
> This seems like a big claim, I certainly haven't seen that on my setup:
>
> * Before:
> $ rm -fr tmp cache
> real    0m14.751s
> user    0m0.323s
> sys     0m0.048s
>
> * After:
> $ rm -fr tmp cache ; time bitbake -p
> real    0m14.725s
> user    0m0.326s
> sys     0m0.046s
>
> but it's still a sensible change.  When I ran a test before/after
> configuration for augeas the configuration seemed off, though.  Can you
> confirm that with this change as is you're getting the correct
> --with/--without and --enable/--disable and patches applied for your
> layers?  I just want to confirm since the ~20s difference in parsing
> seems kind of out of scale for moving essentially three bbappends around
> and I'm wondering if there's something else siginficant in your tree we
> want to consider.


This patch is from Robert Yang. CC to him. Maybe he can give us more 
explanation.

For the augeas, the current augeas_%.bbapend doesn't work because the 
augeas recipe is in meta-oe layer but not meta-python layer. This patch 
moves the bbappend to the correct layer to fix this issue.

It works on my local:

$ cat log.do_configure

[snip]
checking for library containing setfilecon... -lselinux
[snip]
checking for selinux/selinux.h... (cached) yes
checking selinux/context.h usability... yes
checking selinux/context.h presence... yes
checking for selinux/context.h... yes
[snip]


//Yi


>
> -J.
>
>> Signed-off-by: Robert Yang <liezhi.yang at windriver.com>
>> Signed-off-by: Yi Zhao <yi.zhao at windriver.com>
>> ---
>>   conf/layer.conf                                               | 11 +++++++----
>>   .../recipes-daemons/iscsi-initiator-utils/files/initd.debian  |  0
>>   .../iscsi-initiator-utils/iscsi-initiator-utils_%.bbappend    |  0
>>   .../iscsi-initiator-utils/iscsi-initiator-utils_selinux.inc   |  0
>>   .../recipes-support}/augeas/augeas_%.bbappend                 |  0
>>   .../recipes-containers/lxc/lxc_%.bbappend                     |  0
>>   6 files changed, 7 insertions(+), 4 deletions(-)
>>   rename {networking-layer => dynamic-layers/networking-layer}/recipes-daemons/iscsi-initiator-utils/files/initd.debian (100%)
>>   rename {networking-layer => dynamic-layers/networking-layer}/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_%.bbappend (100%)
>>   rename {networking-layer => dynamic-layers/networking-layer}/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_selinux.inc (100%)
>>   rename {meta-python/recipes-extended/augeas => dynamic-layers/openembedded-layer/recipes-support}/augeas/augeas_%.bbappend (100%)
>>   rename {virtualization-layer => dynamic-layers/virtualization-layer}/recipes-containers/lxc/lxc_%.bbappend (100%)
>>
>> diff --git a/conf/layer.conf b/conf/layer.conf
>> index 9dd34b1..89b9468 100644
>> --- a/conf/layer.conf
>> +++ b/conf/layer.conf
>> @@ -5,10 +5,13 @@ BBPATH .= ":${LAYERDIR}"
>>   BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
>>               ${LAYERDIR}/recipes-*/*/*.bbappend"
>>   
>> -# Let us add layer-specific bbappends which are only applied when that
>> -# layer is included in our configuration
>> -BBFILES += "${@' '.join('${LAYERDIR}/%s/recipes*/*/*.bbappend' % layer \
>> -               for layer in BBFILE_COLLECTIONS.split())}"
>> +BBFILES_DYNAMIC += "openembedded-layer:${LAYERDIR}/dynamic-layers/openembedded-layer/*/*/*.bb \
>> +                    openembedded-layer:${LAYERDIR}/dynamic-layers/openembedded-layer/*/*/*.bbappend \
>> +                    networking-layer:${LAYERDIR}/dynamic-layers/networking-layer/*/*/*.bb \
>> +                    networking-layer:${LAYERDIR}/dynamic-layers/networking-layer/*/*/*.bbappend \
>> +                    virtualization-layer:${LAYERDIR}/dynamic-layers/virtualization-layer/recipes*/*/*.bb \
>> +                    virtualization-layer:${LAYERDIR}/dynamic-layers/virtualization-layer/recipes*/*/*.bbappend \
>> +                   "
>>   
>>   BBFILE_COLLECTIONS += "selinux"
>>   BBFILE_PATTERN_selinux = "^${LAYERDIR}/"
>> diff --git a/networking-layer/recipes-daemons/iscsi-initiator-utils/files/initd.debian b/dynamic-layers/networking-layer/recipes-daemons/iscsi-initiator-utils/files/initd.debian
>> similarity index 100%
>> rename from networking-layer/recipes-daemons/iscsi-initiator-utils/files/initd.debian
>> rename to dynamic-layers/networking-layer/recipes-daemons/iscsi-initiator-utils/files/initd.debian
>> diff --git a/networking-layer/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_%.bbappend b/dynamic-layers/networking-layer/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_%.bbappend
>> similarity index 100%
>> rename from networking-layer/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_%.bbappend
>> rename to dynamic-layers/networking-layer/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_%.bbappend
>> diff --git a/networking-layer/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_selinux.inc b/dynamic-layers/networking-layer/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_selinux.inc
>> similarity index 100%
>> rename from networking-layer/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_selinux.inc
>> rename to dynamic-layers/networking-layer/recipes-daemons/iscsi-initiator-utils/iscsi-initiator-utils_selinux.inc
>> diff --git a/meta-python/recipes-extended/augeas/augeas/augeas_%.bbappend b/dynamic-layers/openembedded-layer/recipes-support/augeas/augeas_%.bbappend
>> similarity index 100%
>> rename from meta-python/recipes-extended/augeas/augeas/augeas_%.bbappend
>> rename to dynamic-layers/openembedded-layer/recipes-support/augeas/augeas_%.bbappend
>> diff --git a/virtualization-layer/recipes-containers/lxc/lxc_%.bbappend b/dynamic-layers/virtualization-layer/recipes-containers/lxc/lxc_%.bbappend
>> similarity index 100%
>> rename from virtualization-layer/recipes-containers/lxc/lxc_%.bbappend
>> rename to dynamic-layers/virtualization-layer/recipes-containers/lxc/lxc_%.bbappend
>> -- 
>> 2.7.4
>>