[yocto] RFC: Backwards compatibility checking sstate-cache

Darcy Watkins dwatkins at sierrawireless.com
Fri Jun 30 08:01:12 PDT 2017


Hi,

It would also be nice to add some sort of hook script support where user supplied hook script performs a check and returns either a "yes it's OK" or a "no have to rebuild it" status.
 And then tie that into the dependency logic.

I have unusual use case where I need to have a dependency on changes to the kernel version, but not necessarily something triggered after every kernel rebuild.   The package I'm building will trigger OpenSSL and a whole whack of stuff each time it rebuilds.

It depends on the kernel version and the headers, but shouldn't need to depend on every patch or kconfig change that happens.


___

Regards,

Darcy

Darcy Watkins ::  Senior Staff Engineer, Firmware

SIERRA WIRELESS
Direct  +1 604 233 7989<tel:+1%20604%20233%207989>   ::  Fax  +1 604 231 1109<tel:+1%20604%20231%201109>  ::  Main  +1 604 231 1100<tel:+1%20604%20231%201100>
13811 Wireless Way  :: Richmond, BC Canada V6V 3A4<x-apple-data-detectors://3/1>
[M4]
dwatkins at sierrawireless.com<mailto:dwatkins at sierrawireless.com> :: www.sierrawireless.com<http://www.sierrawireless.com/> :: www.inmotiontechnology.com<http://www.inmotiontechnology.com/>


On Jun 30, 2017, at 2:53 AM, Michael Ho <Michael.Ho at bmw-carit.de<mailto:Michael.Ho at bmw-carit.de>> wrote:

Hi, at BMW Car IT we are working on an experimental feature to improve sstate
cache hits and we are looking for comments on the approach who might have some
insights to the problem and seeing if anyone is interested in the feature for
mainline.

The sstate-cache of a recipe is tied closely to its build dependencies, as the
signature generated for a task includes all parent task's signatures as part of
the signature information. This means that when changes occur in the parent
recipes, even if insignificant, all children recipes that have valid sstate
cache must invalidate their sstate cache objects.

What this patchset does is propose to add another optional variable to recipes,
CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature
generation it excludes any parent tasks that come from dependencies listed in
it. This is to break the signature change domino effect.

This patchset also proposes modifying RunQueue to then be able to run a
compatibility checker during task execution phase for recipes and tasks that use
CDEPENDS and allow for deciding to re-execute a task despite being covered by
sstate-cache.

The patchset is based on the jethro branch for the poky repository, as this is
the branch that we are using.  If the general idea sounds good, we can consider
porting it to master.

Included is an patch that adds an example recipe and compatibility checker,
where compatibility is based on the file checksums of the parent recipes
packages. An example recipe, cdepends-test1, generates a compatibility report
containing the file checksums of all files that it packages and which file paths
they are at. Another recipe, cdepends-test2, can then strip this compatibility
report to the minimal files it needs to be consistent and can compare the latest
checksums it used to configure/compile/install with and if they have changed,
trigger a rebuild. If not, the previous version restored from sstate-cache is
used.

We are still experimenting with the usages of this, including the use of having
abi-compliance-checker to compare the ABI of shared libraries as a compatibility
checker during RunQueue and using the results to avoid rebuilding child recipes
when the .so files they depend on are still compatible. Example use cases of
this are allowing recipes which provide multiple shared libraries to change a
single .so file without rebuilding all its children that depend on the other
shared libraries but not the one that changed.

We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it didn't meet
the feature requirements of what this compatibility checker callback is doing,
although maybe when porting to master we could refactor to make better use of
the work already done there. The current implementation is a bit hacky but
comments would be appreciated in regards to if the concept is feasible and if
people are interested in making use of it and their use cases.

Kind regards,
Michael Ho

--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm

Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael.ho at bmw-carit.de<mailto:michael.ho at bmw-carit.de>
Web: http://www.bmw-carit.de
--------------------------------------------------------
BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--------------------------------------------------------
<0001-cache.py-add-support-for-CDEPENDS.patch>
<0002-siggen.py-add-support-for-CDEPENDS.patch>
<0003-runqueue.py-add-support-for-CDEPENDS.patch>
<0004-taskdata.py-add-support-for-CDEPENDS.patch>
<0005-rm_work.bbclass-add-exception-for-do_cache_compatibi.patch>
<0006-native.bbclass-add-support-for-CDEPENDS.patch>
<0007-nativesdk.bbclass-add-support-for-CDEPENDS.patch>
<0008-classes-add-bbclasses-compatible-depends-and-compati.patch>
<0009-recipes-cdepends-test-example-recipe-framework-for-u.patch>
--
_______________________________________________
yocto mailing list
yocto at yoctoproject.org<mailto:yocto at yoctoproject.org>
https://lists.yoctoproject.org/listinfo/yocto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20170630/a40eee1b/attachment.html>


More information about the yocto mailing list