[yocto] RFC: Backwards compatibility checking sstate-cache

Fri Sep 22 07:00:49 PDT 2017

I think this remark in the referenced link is the best summary of "what could 
be improved":

"""the biggest weakness of the sstate signature bits, in my opinion, is that 
it only tracks inputs, not outputs. If task A depends on B, and the metadata 
input to B changes, then A will be rebuilt, even if the *output* of B didn't 
change as a result of the change to its metadata."""

For example, if someone fixes a bug in gcc that only applies to MIPS, then 
builds that only target an ARM machine shouldn't be affected. Much worse than 
that, I have a dependency like:
gcc -> ... -> python -> bitstream tool -> FPGA image

so this means that a change in gcc causes Python to rebuild, which causes the 
bitstream tool to be rebuilt and produce idential output, and that triggers a 
roughly 31-hour build of lots of FPGA images. These are the ones we really 
want to break. A binary output compare would help a lot, even if 80% of the 
libraries fail to create reproducible output, I may still be spared those 31 
hours...

I think tracking digital output is technically feasible, and won't require any 
change to existing recipes.

Also think about "feed" setups. When I see my settop reporting "331 packages 
need updating"... It'd be great if that could be avoided.

On 01-07-17 09:48, Martin Jansa wrote:
> I haven't tried the patches, but I really like this idea (I was suggesting 
> something like that since 2011 
> http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/10327) and 
> I'm glad you weren't discouraged attempting to do this.
> 
> It also implements 3) b) idea from 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970 , you might be 
> interested read that ticket.
> 
> Thanks
> 
> On Fri, Jun 30, 2017 at 11:48 AM, Michael Ho <Michael.Ho at bmw-carit.de 
> <mailto:Michael.Ho at bmw-carit.de>> wrote:
> 
>     Hi, at BMW Car IT we are working on an experimental feature to improve sstate
>     cache hits and we are looking for comments on the approach who might have some
>     insights to the problem and seeing if anyone is interested in the feature for
>     mainline.
> 
>     The sstate-cache of a recipe is tied closely to its build dependencies, as the
>     signature generated for a task includes all parent task's signatures as
>     part of
>     the signature information. This means that when changes occur in the parent
>     recipes, even if insignificant, all children recipes that have valid sstate
>     cache must invalidate their sstate cache objects.
> 
>     What this patchset does is propose to add another optional variable to
>     recipes,
>     CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature
>     generation it excludes any parent tasks that come from dependencies listed in
>     it. This is to break the signature change domino effect.
> 
>     This patchset also proposes modifying RunQueue to then be able to run a
>     compatibility checker during task execution phase for recipes and tasks
>     that use
>     CDEPENDS and allow for deciding to re-execute a task despite being covered by
>     sstate-cache.
> 
>     The patchset is based on the jethro branch for the poky repository, as this is
>     the branch that we are using.  If the general idea sounds good, we can
>     consider
>     porting it to master.
> 
>     Included is an patch that adds an example recipe and compatibility checker,
>     where compatibility is based on the file checksums of the parent recipes
>     packages. An example recipe, cdepends-test1, generates a compatibility report
>     containing the file checksums of all files that it packages and which file
>     paths
>     they are at. Another recipe, cdepends-test2, can then strip this compatibility
>     report to the minimal files it needs to be consistent and can compare the
>     latest
>     checksums it used to configure/compile/install with and if they have changed,
>     trigger a rebuild. If not, the previous version restored from sstate-cache is
>     used.
> 
>     We are still experimenting with the usages of this, including the use of
>     having
>     abi-compliance-checker to compare the ABI of shared libraries as a
>     compatibility
>     checker during RunQueue and using the results to avoid rebuilding child
>     recipes
>     when the .so files they depend on are still compatible. Example use cases of
>     this are allowing recipes which provide multiple shared libraries to change a
>     single .so file without rebuilding all its children that depend on the other
>     shared libraries but not the one that changed.
> 
>     We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it
>     didn't meet
>     the feature requirements of what this compatibility checker callback is doing,
>     although maybe when porting to master we could refactor to make better use of
>     the work already done there. The current implementation is a bit hacky but
>     comments would be appreciated in regards to if the concept is feasible and if
>     people are interested in making use of it and their use cases.
> 
>     Kind regards,
>     Michael Ho
> 
>     --
>     BMW Car IT GmbH
>     Michael Ho
>     Spezialist Entwicklung - Linux Software Integration
>     Lise-Meitner-Str. 14
>     89081 Ulm
> 
>     Tel.: +49 731 3780 4071 <tel:%2B49%20731%203780%204071>
>     Mobil: +49 152 5498 0471 <tel:%2B49%20152%205498%200471>
>     Fax: +49-731-37804-001 <tel:%2B49-731-37804-001>
>     Mail: michael.ho at bmw-carit.de <mailto:michael.ho at bmw-carit.de>
>     Web: http://www.bmw-carit.de
>     --------------------------------------------------------
>     BMW Car IT GmbH
>     Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
>     Sitz und Registergericht: München HRB 134810
>     --------------------------------------------------------
> 
>     --
>     

Kind regards,

Mike Looijmans
System Expert

TOPIC Products
Materiaalweg 4, NL-5681 RJ Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijmans at topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Visit us at the Space Tech Expo Europe (Stand E-71)
_______________________________________________
>     yocto mailing list
>     yocto at yoctoproject.org <mailto:yocto at yoctoproject.org>
>     https://lists.yoctoproject.org/listinfo/yocto
>     <https://lists.yoctoproject.org/listinfo/yocto>
> 
> 
> 
> 