[Automated-testing] [Openembedded-architecture] Yocto Project LTS Proposal/Discussion

[Richard Purdie]

On Sat, 2019-10-26 at 10:34 -0400, Rich Persaud wrote:
> On Oct 26, 2019, at 7:11 AM, richard.purdie at linuxfoundation.org
> wrote:
> > On Fri, 2019-10-25 at 16:04 -0400, Rich Persaud wrote:
> > > There may be lessons in Microsoft's move from diverse hardware to
> > > VM-
> > > based testing:
> > > https://www.ghacks.net/2019/09/23/former-microsoft-employee-explains-why-bugs-in-windows-updates-increased/
> > 
> > I don't disagreed, real hardware does have some distinct
> > advantages. We
> > also have to keep in mind that we're testing what the Yocto Project
> > is
> > offering. At some point YP has to trust that the underlying kernels
> > we
> > ship are "ok" and that upstream has covered some of this.
> 
> Here's a quote from John Loucaides of Eclypsium, ex-Intel, 
> https://twitter.com/johnloucaides/status/1187839034661339136,
> describing his PSEC video:
> 
> "While hard, I don't believe supply chain security is unsolvable. I
> believe it's a "tragedy of the commons" problem where everyone hopes
> someone else will fix it. By working together on tools/projects, we
> can change the incentives and create practical solutions."

Sure.

As a project we struggle to do what we do with the resources we have
available today, to the point that we're burning out people, me
included.

I have pushed and tried to have automated real hardware testing for
over a decade. Its proving very difficult to get traction and make it
happen.

We could make a significant difference under virtualised testing so
we've done that, I've spent a considerable amount of time ensuring YP
at least has robust testing there. There is much more I'd love to do
but I personally don't have the hours in the day and I struggle to find
people to help.

I'm a realist. I'd like YP LTS to happen. The bigger the barriers we
put in its way, the less likely it is. I've always aimed to build
systems and technologies that do allow incremental
development/extension so there is at least options for this.

> > Virtual testing works well for the software stack itself which a
> > large
> > part of what YP offers, particularly where we can run upstream's
> > tests
> > for components. Where its struggles is on the variations of
> > hardware
> > that are out there as the article highlights.
> > 
> > As such, I think virtual testing for any YP LTS is the realistic
> > option
> > we have right now.
> 
> Some upstream kernel testing is also done on virtual machines, for
> similar reasons. They may expect downstream "distros" to have more
> device-focused hardware coverage. 
> 
> To avoid YP users assuming that YP and/or the upstream kernel has
> done hardware testing for YP LTS, we may want to document specific
> testing actions that are expected from YP LTS users.  E.g. should YP
> downstreams work directly with CKI via the Linux Foundation, to pool
> hardware test results?  Should they report hw test results to other
> YP LTS users?

We do document very clearly the testing that YP has done for each
release. Users are already expected to figure out and understand what
additional testing their end use case may need and LTS is no different.
Their hardware for example will look very different, even between
users.

Pooling results works if the base configurations can be summarised and
reproduced. I know the automated-testing people are working on ways to
improve the ability to do that. For the YP side we do have some helpful
technology for example the hash representations of our configurations
which cover most of the software supply chain.

> > > Business requirements for "LTS releases" include security fixes
> > > and
> > > non-regression of production systems, i.e. the software supply
> > > chain
> > > security topics discussed in [1] above.  There were CKI [3]
> > > discussions about sharing test resources (human, machine) for
> > > kernel
> > > testing.  Some CKI challenges and potential solutions [4] may be
> > > applicable to Yocto LTS testing of packages beyond the Linux
> > > kernel.
> > > VM-based testing could be supplemented with pooled test results
> > > from
> > > distributed testing on diverse hardware, across the Yocto
> > > contributor
> > > community.  Even with VM-based testing, IOMMU PCI passthrough can
> > > be
> > > used to give a VM direct access to a PCI device, for device
> > > driver
> > > testing.
> > 
> > We have already build a mechanism into our test results process for
> > externally contributed test results and we are happy to collect
> > that
> > data although we haven't seen much being contributed yet. I'd love
> > to
> > see more data coming in for any LTS though (or any other release
> > for
> > that matter).
> 
> That's good news.  Could you share pointers/docs/samples on the
> mechanism for external test result contributions?

I've worked on material which I gave to our tech writer but the
resulting docs haven't been produced yet which comes down to resources
:(.

I've also posted many times over the last year about the Yocto QA
developments, how the new process works and how people could contribute
test results to that.

The QA page on the wiki:

https://wiki.yoctoproject.org/wiki/QA

has the basics, its not extensively documented but its not a complex
process either and it is the process Intel/WindRiver use to contribute
test results today.

>   Is this effort coordinated with ATS or CKI for pooling of hw-
> specific test results?

No, its hard enough to get YP testing to work without more parties
involved. I was actually specifically told by managers to even drop the
generic process as it was adding too much overhead and complexity. I
refused, despite significant pressure FWIW.

>   If we are able to attract contributions, could we eventually
> associate YP LTS external test results with public BSP definitions
> and hardware model/revision numbers?

I'd love to see it happen and I do understand and like your vision but
sadly companies don't seem to see the value of putting resources into
things like this.

Cheers,

Richard