[Automated-testing] [Openembedded-architecture] Yocto Project LTS Proposal/Discussion

[Rich Persaud]

On Oct 26, 2019, at 11:45 AM, Richard Purdie <richard.purdie at linuxfoundation.org> wrote:
> 
> On Sat, 2019-10-26 at 10:34 -0400, Rich Persaud wrote:
>>> On Oct 26, 2019, at 7:11 AM, richard.purdie at linuxfoundation.org
>>> wrote:
>>> On Fri, 2019-10-25 at 16:04 -0400, Rich Persaud wrote:
>>>> There may be lessons in Microsoft's move from diverse hardware to
>>>> VM-
>>>> based testing:
>>>> https://www.ghacks.net/2019/09/23/former-microsoft-employee-explains-why-bugs-in-windows-updates-increased/
>>> I don't disagreed, real hardware does have some distinct
>>> advantages. We
>>> also have to keep in mind that we're testing what the Yocto Project
>>> is
>>> offering. At some point YP has to trust that the underlying kernels
>>> we
>>> ship are "ok" and that upstream has covered some of this.
>> Here's a quote from John Loucaides of Eclypsium, ex-Intel,
>> https://twitter.com/johnloucaides/status/1187839034661339136,
>> describing his PSEC video:
>> "While hard, I don't believe supply chain security is unsolvable. I
>> believe it's a "tragedy of the commons" problem where everyone hopes
>> someone else will fix it. By working together on tools/projects, we
>> can change the incentives and create practical solutions."
> 
> Sure.
> 
> As a project we struggle to do what we do with the resources we have
> available today, to the point that we're burning out people, me
> included.
> 
> I have pushed and tried to have automated real hardware testing for
> over a decade. Its proving very difficult to get traction and make it
> happen.
> 
> We could make a significant difference under virtualised testing so
> we've done that, I've spent a considerable amount of time ensuring YP
> at least has robust testing there. There is much more I'd love to do
> but I personally don't have the hours in the day and I struggle to find
> people to help.
> 
> I'm a realist. I'd like YP LTS to happen. The bigger the barriers we
> put in its way, the less likely it is. I've always aimed to build
> systems and technologies that do allow incremental
> development/extension so there is at least options for this.
> 
>>> Virtual testing works well for the software stack itself which a
>>> large
>>> part of what YP offers, particularly where we can run upstream's
>>> tests
>>> for components. Where its struggles is on the variations of
>>> hardware
>>> that are out there as the article highlights.
>>> As such, I think virtual testing for any YP LTS is the realistic
>>> option
>>> we have right now.
>> Some upstream kernel testing is also done on virtual machines, for
>> similar reasons. They may expect downstream "distros" to have more
>> device-focused hardware coverage.
>> To avoid YP users assuming that YP and/or the upstream kernel has
>> done hardware testing for YP LTS, we may want to document specific
>> testing actions that are expected from YP LTS users.  E.g. should YP
>> downstreams work directly with CKI via the Linux Foundation, to pool
>> hardware test results?  Should they report hw test results to other
>> YP LTS users?
> 
> We do document very clearly the testing that YP has done for each
> release. Users are already expected to figure out and understand what
> additional testing their end use case may need and LTS is no different.
> Their hardware for example will look very different, even between
> users.
> 
> Pooling results works if the base configurations can be summarised and
> reproduced. I know the automated-testing people are working on ways to
> improve the ability to do that. For the YP side we do have some helpful
> technology for example the hash representations of our configurations
> which cover most of the software supply chain.
> 
>>>> Business requirements for "LTS releases" include security fixes
>>>> and
>>>> non-regression of production systems, i.e. the software supply
>>>> chain
>>>> security topics discussed in [1] above.  There were CKI [3]
>>>> discussions about sharing test resources (human, machine) for
>>>> kernel
>>>> testing.  Some CKI challenges and potential solutions [4] may be
>>>> applicable to Yocto LTS testing of packages beyond the Linux
>>>> kernel.
>>>> VM-based testing could be supplemented with pooled test results
>>>> from
>>>> distributed testing on diverse hardware, across the Yocto
>>>> contributor
>>>> community.  Even with VM-based testing, IOMMU PCI passthrough can
>>>> be
>>>> used to give a VM direct access to a PCI device, for device
>>>> driver
>>>> testing.
>>> We have already build a mechanism into our test results process for
>>> externally contributed test results and we are happy to collect
>>> that
>>> data although we haven't seen much being contributed yet. I'd love
>>> to
>>> see more data coming in for any LTS though (or any other release
>>> for
>>> that matter).
>> That's good news.  Could you share pointers/docs/samples on the
>> mechanism for external test result contributions?
> 
> I've worked on material which I gave to our tech writer but the
> resulting docs haven't been produced yet which comes down to resources
> :(.
> 
> I've also posted many times over the last year about the Yocto QA
> developments, how the new process works and how people could contribute
> test results to that.
> 
> The QA page on the wiki:
> 
> https://wiki.yoctoproject.org/wiki/QA
> 
> has the basics, its not extensively documented but its not a complex
> process either and it is the process Intel/WindRiver use to contribute
> test results today.
> 
>> Is this effort coordinated with ATS or CKI for pooling of hw-
>> specific test results?
> 
> No, its hard enough to get YP testing to work without more parties
> involved. I was actually specifically told by managers to even drop the
> generic process as it was adding too much overhead and complexity. I
> refused, despite significant pressure FWIW.
> 
>> If we are able to attract contributions, could we eventually
>> associate YP LTS external test results with public BSP definitions
>> and hardware model/revision numbers?
> 
> I'd love to see it happen and I do understand and like your vision but
> sadly companies don't seem to see the value of putting resources into
> things like this.


Some companies have evolved their business models to monetize complex (OSS + proprietary) process, silicon, software, model, and data supply chains.

kernelci.org (announced today [1]) is now part of Linux Foundation, joining LVFS (Linux firmware updates), CCC (confidential computing hardware enclave SDK) and CHIPS Alliance (OSS EDA tools for open hardware).

Google helped drive upstream Linux kernel LTS, motivated by the economics of Android support.  Microsoft Linux has a license that includes 10 years (!) of security updates.  Both are contributing to kernelci.org, along with RedHat's CKI.

PSEC video [2] for Azure Sphere, and the final slide in the linked PDF, mentions nightly automated testing, with static analysis and fuzzing. "IoT OS" is based on Yocto.   There's also a linked talk by Ryan Fairfax with more detail on the build system.

Rich

[1] http://tradeshownews.vporoom.com/2019-10-28-Distributed-Linux-Testing-Platform-KernelCI-Secures-Funding-and-Long-Term-Sustainability-as-New-Linux-Foundation-Project

[2] https://www.platformsecuritysummit.com/2019/speaker/seay/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/automated-testing/attachments/20191028/f80991cb/attachment-0001.html>