[Automated-testing] [Openembedded-architecture] Yocto Project LTS Proposal/Discussion

[Rich Persaud]

On Oct 24, 2019, at 20:08, Rich Persaud <persaur at gmail.com> wrote:
> 
> (adding the automated-testing list)
> 
>> On Oct 24, 2019, at 18:52, Richard Purdie <richard.purdie at linuxfoundation.org> wrote:
>> I'm posting this to openembedded-architecture since it probably is the
>> best place for discussions about OE/Yocto planning.
>> 
>> The Yocto Project TSC believes one of the things needed for YP and for
>> OE is more information being pulled together about how an LTS release
>> could work. The document linked to below is the information it was able
>> to collate together on the subject. Its the result of some long
>> discussions by the TSC.
>> ...
>> The key thing needed to make this happen is a resource commitment to
>> it. Without such a commitment it is simply not feasible.
> 
> Vendors who provide commercial support for OE/Yocto-based products may already be investing downstream resources for equivalent "LTS" support.  May be useful for the document to show the financial benefits of shifting a percentage of downstream LTS validation resources to upstream Yocto LTS.
> 
>> There may be
>> new companies willing to join the project in order to make it happen,
>> if so we should talk further.
>> 
>> The document:
>> 
>> https://docs.google.com/document/d/1AwAFDf52f_FoXksbHEVUMlu4hpcI0JMGVG-Kj_sUkyc/
> 
> > The project components to be covered would need to match those included in our standard release process and should be clearly defined. Those components would be:
> > Bitbake
> > OE-Core
> > Meta-yocto
> > yocto-docs
> ...
> > For simplicity, it is also proposed that only automated testing be used for testing the LTS and that any current manual QA is not performed.
> ...
> > Only run virtualized tests
> > Only support one host distro, an Ubuntu LTS as it has right lifespan
> 
> When the first Yocto LTS release is available, could that become the host distro for automated testing?  That would provide Yocto automated testing with supply chain integrity benefits from Yocto.  If we take that path, with automated testing based on virtualization, could we add meta-virtualization to the list of Yocto LTS release components?
> 
> Yocto-on-Yocto automated testing could reduce risks [1] that impact software supply chains. A potential topic for the ATS event [2] next week.
> 
> Rich
> 
> [1] https://www.platformsecuritysummit.com/2019/speaker/sherman/
> 
> [2] https://elinux.org/Automated_Testing_Summit_2019

There may be lessons in Microsoft's move from diverse hardware to VM-based testing:
https://www.ghacks.net/2019/09/23/former-microsoft-employee-explains-why-bugs-in-windows-updates-increased/

> Back in 2014/2015, Microsoft employed an entire team that was dedicated to testing the operating system, builds, updates, drivers, and other code. The team consisted of multiple groups that would run tests and discuss bugs and issues in daily meetings. Tests were conducted manually by the team and through automated testing, and if tests were passed, would give the okay to integrate the code into Windows.
> 
> The teams ran the tests on "real" hardware in a lab through automated testing. The machines had different hardware components, e.g. processors, hard drives, video and sound cards, and other components to cover a wide range of system configurations, and this meant that bugs that affected only certain hardware components or configurations were detected in the process.
> 
> Microsoft laid off almost the entire Windows Test team as it moved the focus from three different systems -- Windows, Windows Mobile and Xbox -- to a single system. The company moved most of the testing to virtual machines and this meant according to Berg that tests were no longer conducted on real and diverse hardware configurations for the most part.

Business requirements for "LTS releases" include security fixes and non-regression of production systems, i.e. the software supply chain security topics discussed in [1] above.  There were CKI [3] discussions about sharing test resources (human, machine) for kernel testing.  Some CKI challenges and potential solutions [4] may be applicable to Yocto LTS testing of packages beyond the Linux kernel.

VM-based testing could be supplemented with pooled test results from distributed testing on diverse hardware, across the Yocto contributor community.  Even with VM-based testing, IOMMU PCI passthrough can be used to give a VM direct access to a PCI device, for device driver testing.

With firmware attacks, the behavior of hardware-under-test could be altered between test runs, which may invalidate test results and certification claims that identify a DUT.  There are "boot integrity" projects with tamper detection and firmware measurements on TPM-enabled hardware. These hashes can be recorded [5] alongside test results, when using an OS with build and boot integrity on test and logging systems.

Rich

[3] https://cki-project.org/
[4] https://docs.google.com/document/d/1EIU-GEJpChfB2TLzi3ebXQqUnXQ1CQ2gyl48FE-dfQI
[5] https://platformsecuritysummit.com/2019/speaker/loucaides

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/automated-testing/attachments/20191025/914323e5/attachment.html>