[Automated-testing] [Openembedded-architecture] Yocto Project LTS Proposal/Discussion

[richard.purdie at linuxfoundation.org]

On Fri, 2019-10-25 at 16:04 -0400, Rich Persaud wrote:
> There may be lessons in Microsoft's move from diverse hardware to VM-
> based testing:
> https://www.ghacks.net/2019/09/23/former-microsoft-employee-explains-why-bugs-in-windows-updates-increased/

I don't disagreed, real hardware does have some distinct advantages. We
also have to keep in mind that we're testing what the Yocto Project is
offering. At some point YP has to trust that the underlying kernels we
ship are "ok" and that upstream has covered some of this.

Virtual testing works well for the software stack itself which a large
part of what YP offers, particularly where we can run upstream's tests
for components. Where its struggles is on the variations of hardware
that are out there as the article highlights.

As such, I think virtual testing for any YP LTS is the realistic option
we have right now.

> Business requirements for "LTS releases" include security fixes and
> non-regression of production systems, i.e. the software supply chain
> security topics discussed in [1] above.  There were CKI [3]
> discussions about sharing test resources (human, machine) for kernel
> testing.  Some CKI challenges and potential solutions [4] may be
> applicable to Yocto LTS testing of packages beyond the Linux kernel.
> 
> VM-based testing could be supplemented with pooled test results from
> distributed testing on diverse hardware, across the Yocto contributor
> community.  Even with VM-based testing, IOMMU PCI passthrough can be
> used to give a VM direct access to a PCI device, for device driver
> testing.

We have already build a mechanism into our test results process for
externally contributed test results and we are happy to collect that
data although we haven't seen much being contributed yet. I'd love to
see more data coming in for any LTS though (or any other release for
that matter).

Cheers,

Richard