[yocto] QA cycle report for 2.6 M2 RC1

[Yeoh, Ee Peng]

Hi Richard,

There are 23 tests were not executed. 
13 of these tests were related to SDK automated tests (part of it belong to galculator test).
6 of these tests (selftest/cases/distrodata.py), where it was configured to be skipped by Autobuilder (we will removed these tests from planned tests starting new cycle).
2 of these tests (runtime/cases/oe_syslog.py), were skipped due to tests can't run with sysklogd installed.
1 of the runtime/xorg tests running on qemuarm64 were not executed by the testimage. 

For the eSDK automated tests issue, we had filed for bug.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12875

Please let me know if any more question.

Thanks,
Yeoh Ee Peng 

-----Original Message-----
From: richard.purdie at linuxfoundation.org [mailto:richard.purdie at linuxfoundation.org] 
Sent: Thursday, August 9, 2018 4:29 PM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; 'yocto at yoctoproject.org' <yocto at yoctoproject.org>; Jolley, Stephen K <stephen.k.jolley at intel.com>; Eggleton, Paul <paul.eggleton at intel.com>
Cc: Sangal, Apoorv <apoorv.sangal at intel.com>; Kirkiris, Nectar <nectar.kirkiris at intel.com>
Subject: Re: QA cycle report for 2.6 M2 RC1

On Thu, 2018-08-09 at 05:57 +0000, Yeoh, Ee Peng wrote:
> ======= Summary ========
>  
> All planned tests were executed except a few SDK automated tests were 
> skipped in Autobuilder (testrun# 9830-9836, 9796-9797).  Team found 
> that these SDK automated tests were running successfully in core- 
> image-sato-sdk image but not the existing core-image-sato used by 
> Autobuilder, investigating why these automated tests were skipped in 
> core-image-sato and no message available in testsdk logfile to explain 
> why it was skipped.

I think Ross just sent a patch for this, assuming it was the galculator test?

>   Team also found that eSDK devtool automated testcase(s) were skipped 
> unexpectedly due to the unknown error on testcase dependency logic, 
> temporary removing the dependency logic and successfully executed the 
> devtool tests.

We should probably have a bug to track and resolve this issue as it hasn't been resolved yet as far as I know. I believe its a problem introduced by the parallelism changes. We may have to rework the dependency logic.

> New Bugs
> [1] Bug 12866 - [2.6 M2 rc1] wic test_fix_size failed
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12866
>  
> [2] Bug 12864 - [2.6 M2 rc1] buildhistorydifftest failed
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12864
>  
> [3] Bug 12869 - [2.6 M2 rc1] libxml test cases failed
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12869

The good news is that these have all already been fixed in master and are understood issues with good resolutions. I don't believe any of them are enough to warrant an rc2. My recommendation is we release rc1 subject to my question below.

Why does the QA report not show 100% of tests were completed in all cases?

We do need to take the list of bugs in the QA report itself and see if we can resolve some of them for the final release, Stephen, your help in driving/tracking that would be appreciated.

Cheers,

Richard