[Automated-testing] [Openembedded-architecture] Yocto Project LTS Proposal/Discussion

[Rich Persaud]

On Oct 26, 2019, at 7:11 AM, richard.purdie at linuxfoundation.org wrote:
> 
> On Fri, 2019-10-25 at 16:04 -0400, Rich Persaud wrote:
>> There may be lessons in Microsoft's move from diverse hardware to VM-
>> based testing:
>> https://www.ghacks.net/2019/09/23/former-microsoft-employee-explains-why-bugs-in-windows-updates-increased/
> 
> I don't disagreed, real hardware does have some distinct advantages. We
> also have to keep in mind that we're testing what the Yocto Project is
> offering. At some point YP has to trust that the underlying kernels we
> ship are "ok" and that upstream has covered some of this.

Here's a quote from John Loucaides of Eclypsium, ex-Intel, https://twitter.com/johnloucaides/status/1187839034661339136, describing his PSEC video:

"While hard, I don't believe supply chain security is unsolvable. I believe it's a "tragedy of the commons" problem where everyone hopes someone else will fix it. By working together on tools/projects, we can change the incentives and create practical solutions."

> Virtual testing works well for the software stack itself which a large
> part of what YP offers, particularly where we can run upstream's tests
> for components. Where its struggles is on the variations of hardware
> that are out there as the article highlights.
> 
> As such, I think virtual testing for any YP LTS is the realistic option
> we have right now.

Some upstream kernel testing is also done on virtual machines, for similar reasons. They may expect downstream "distros" to have more device-focused hardware coverage. 

To avoid YP users assuming that YP and/or the upstream kernel has done hardware testing for YP LTS, we may want to document specific testing actions that are expected from YP LTS users.  E.g. should YP downstreams work directly with CKI via the Linux Foundation, to pool hardware test results?  Should they report hw test results to other YP LTS users?


>> Business requirements for "LTS releases" include security fixes and
>> non-regression of production systems, i.e. the software supply chain
>> security topics discussed in [1] above.  There were CKI [3]
>> discussions about sharing test resources (human, machine) for kernel
>> testing.  Some CKI challenges and potential solutions [4] may be
>> applicable to Yocto LTS testing of packages beyond the Linux kernel.
>> 
>> VM-based testing could be supplemented with pooled test results from
>> distributed testing on diverse hardware, across the Yocto contributor
>> community.  Even with VM-based testing, IOMMU PCI passthrough can be
>> used to give a VM direct access to a PCI device, for device driver
>> testing.
> 
> We have already build a mechanism into our test results process for
> externally contributed test results and we are happy to collect that
> data although we haven't seen much being contributed yet. I'd love to
> see more data coming in for any LTS though (or any other release for
> that matter).

That's good news.  Could you share pointers/docs/samples on the mechanism for external test result contributions?  Is this effort coordinated with ATS or CKI for pooling of hw-specific test results?  If we are able to attract contributions, could we eventually associate YP LTS external test results with public BSP definitions and hardware model/revision numbers?

Rich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/automated-testing/attachments/20191026/cc13d85a/attachment-0001.html>