[yocto] RFC: Backwards compatibility checking sstate-cache

[Michael Ho]

Hi Richard (and all),

@Richard: Thanks for the input regarding the topic and apologies for not replying until now, I was a bit buried under workload and simultaneously completing my Masters studies. I also wanted to take some time to digest your comments amid working on this CDEPENDS concept, and I just want to say that I'm very open to finding another way to implement this concept of rebuild avoidance in Yocto.

I think the idea of a non-invasive approach can be a bit tricky, but your idea of a secondary sstate signature and mapping table sounds workable to me. (In fact, as I slowly developed out this concept, I think it's almost required). One point though with the removal of the metadata changes is that the view of "what is compatible" will be concrete and cannot differ among children. Sometimes you have only specific children of a recipe that can handle compatibility changes (and at different levels of compatibility) while others cannot handle any change at all, and that is why I developed with the metadata changes as a basis. But definitely, yes, the change to metadata makes things more complex and does in a sense impose policy, which I understand would be undesirable. Maybe we can figure out an approach that avoids metadata changes but keeps this ability to differ compatibility handling among children.

Currently I have a working proof of concept (I've developed out the earlier shared patches further) that I'd like to share soon and demonstrate. It still needs some further refinement but I think it's showing some interesting results already. I'll think further about the topic and see what I can come up with that matches your line of thoughts.

@Martin Jansa: Thank you for the comments and information, it's good to know that there's some interest from the community to look into this idea.

@Mike Looijmans: Yes, that's essentially the problem I want to resolve with this proof of concept, breaking long build chains at points where a clear precision cut could've been made depending on whether a rebuild is actually required or not (i.e. the bitstream tool not changing). In that case I would even use the tool version and assume even if the tool binary has changed, that the output result of tool would stay the same so it is compatible in essence (and avoid rebuilding 31 hours of stuff).

@Trevor Woerner: Thanks for the suggestion, I was planning to cross-post when I had a more stable proof of concept code to share.

Note: I'll be following my colleague, Mario Domenech Goulart (mario-goulart), to OEDEM 2017 where I hope to discuss the current concept code and show some potential use cases.

Kind regards,
Michael Ho

--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm

Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael.ho at bmw-carit.de
Web: http://www.bmw-carit.de
--------------------------------------------------------
BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--------------------------------------------------------

________________________________________
From: Richard Purdie <richard.purdie at linuxfoundation.org>
Sent: 21 September 2017 18:11
To: Michael Ho; yocto at yoctoproject.org
Subject: Re: [yocto] RFC: Backwards compatibility checking sstate-cache

On Fri, 2017-06-30 at 09:48 +0000, Michael Ho wrote:
> Hi, at BMW Car IT we are working on an experimental feature to
> improve sstate cache hits and we are looking for comments on the
> approach who might have some insights to the problem and seeing if
> anyone is interested in the feature for mainline.

Sorry I didn't see this before now but it was brought to my attention.

I have given this problem quite some thought off and on. I do worry
that the interface you propose is complex and requires changes
throughout the metadata and those changes impose "policy". One of our
strengths is that we're managed to keep "policy" out of the metadata.

In this context by policy, I mean when a recipe is equivalent and when
it is not.

I do have a counter-proposal of how we could solve this in a less
invasive way. This isn't implemented but wouldn't in theory be hard to
do.

The idea would be to have some kind of equivalence list. The first
built object's sstate checksum would become the definitive checksum and
the object added to the sstate cache.

If a new object is built, it would be compared with the one in sstate.
If deemed equivalent (by whatever policy), a mapping would be added to
the list. If not, there is no mapping and it becomes a new definitive
checksum.

All runqueue would then have to do is present its list of sstate
checksums to the list and convert them (in dependency order) into
canonical checksums.

This list could be a local equivalence file, or an equivalence server
which could resolve list of checksums.

Doing it this way totally isolates the comparison from the metadata
itself and in my view protects us from future changes in definition of
equivalence.

How does that sound?

Cheers,

Richard