[yocto] RFC: Backwards compatibility checking sstate-cache

[Martin Jansa]

I haven't tried the patches, but I really like this idea (I was suggesting
something like that since 2011
http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/10327)
and I'm glad you weren't discouraged attempting to do this.

It also implements 3) b) idea from
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970 , you might be
interested read that ticket.

Thanks

On Fri, Jun 30, 2017 at 11:48 AM, Michael Ho <Michael.Ho at bmw-carit.de>
wrote:

> Hi, at BMW Car IT we are working on an experimental feature to improve
> sstate
> cache hits and we are looking for comments on the approach who might have
> some
> insights to the problem and seeing if anyone is interested in the feature
> for
> mainline.
>
> The sstate-cache of a recipe is tied closely to its build dependencies, as
> the
> signature generated for a task includes all parent task's signatures as
> part of
> the signature information. This means that when changes occur in the parent
> recipes, even if insignificant, all children recipes that have valid sstate
> cache must invalidate their sstate cache objects.
>
> What this patchset does is propose to add another optional variable to
> recipes,
> CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for
> signature
> generation it excludes any parent tasks that come from dependencies listed
> in
> it. This is to break the signature change domino effect.
>
> This patchset also proposes modifying RunQueue to then be able to run a
> compatibility checker during task execution phase for recipes and tasks
> that use
> CDEPENDS and allow for deciding to re-execute a task despite being covered
> by
> sstate-cache.
>
> The patchset is based on the jethro branch for the poky repository, as
> this is
> the branch that we are using.  If the general idea sounds good, we can
> consider
> porting it to master.
>
> Included is an patch that adds an example recipe and compatibility checker,
> where compatibility is based on the file checksums of the parent recipes
> packages. An example recipe, cdepends-test1, generates a compatibility
> report
> containing the file checksums of all files that it packages and which file
> paths
> they are at. Another recipe, cdepends-test2, can then strip this
> compatibility
> report to the minimal files it needs to be consistent and can compare the
> latest
> checksums it used to configure/compile/install with and if they have
> changed,
> trigger a rebuild. If not, the previous version restored from sstate-cache
> is
> used.
>
> We are still experimenting with the usages of this, including the use of
> having
> abi-compliance-checker to compare the ABI of shared libraries as a
> compatibility
> checker during RunQueue and using the results to avoid rebuilding child
> recipes
> when the .so files they depend on are still compatible. Example use cases
> of
> this are allowing recipes which provide multiple shared libraries to
> change a
> single .so file without rebuilding all its children that depend on the
> other
> shared libraries but not the one that changed.
>
> We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it
> didn't meet
> the feature requirements of what this compatibility checker callback is
> doing,
> although maybe when porting to master we could refactor to make better use
> of
> the work already done there. The current implementation is a bit hacky but
> comments would be appreciated in regards to if the concept is feasible and
> if
> people are interested in making use of it and their use cases.
>
> Kind regards,
> Michael Ho
>
> --
> BMW Car IT GmbH
> Michael Ho
> Spezialist Entwicklung - Linux Software Integration
> Lise-Meitner-Str. 14
> 89081 Ulm
>
> Tel.: +49 731 3780 4071
> Mobil: +49 152 5498 0471
> Fax: +49-731-37804-001
> Mail: michael.ho at bmw-carit.de
> Web: http://www.bmw-carit.de
> --------------------------------------------------------
> BMW Car IT GmbH
> Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
> Sitz und Registergericht: München HRB 134810
> --------------------------------------------------------
>
> --
> _______________________________________________
> yocto mailing list
> yocto at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20170701/689c72e9/attachment.html>