[yocto] QA Test Report for Yocto Project 2.3 M2 rc1

[Perez Carranza, Jose]

Hi

Here is the report for full QA Cycle on 2.3 M2 rc1

Full Report :  https://wiki.yoctoproject.org/wiki/WW05_-_2017-01-24_-_Full_Test_Cycle_2.3_M2_rc1

Summary

The QA cycle for 2.3 M2 rc1 was completed, 6 new issues were found, the most important was 10964 that is currently blocking all the automated test for BSP module, the biggest module validated during the cycle, the rest of the issues are not representing big blockers. In regards of performance there were important degradations on the tasks measured, mostly caused by "recipe specific sysroots" introduced on this release candidate. For this cycle were added two component to the list of validated items, CROPS and KENREL, both are currently manual executed. Due the Chinese new year festivities we had  a resource missing from WR team hence some components were not validated, mostly the ones related to non x86 arch. There was also an increase of execution time on selftest due combination of current performance degradation and  TC "test_postinst_roofs_and_boot" that is testing different configurations for init/manager packages a patch was sent [1] but still taking considerable AB time due the nature of the test.

Based on the previous statements the QA conclusion is that and rc2  is needed to test automated BSP (testimage),  Selftest and measurements of performance.

Bugs

      * New

10964 - testimage: Test suite is not running when images are downloaded from Autobuilder [2]
10968 - selftest: devtool and recipetool fails on load plugin test. [3]
10970 - pybootchartgui.py has not been migrated to python3 [4]
10983 - [Test Case 810] Avoid cache reload for incremental development using a backgrounded bitbake server [5]
10984 - [Test Case 1058] bitbake can use alternate download methods than git [6]
10990 - [crosstap] systemtap error on core-image-minimal with tools-profile [7]

      * High / M+ Not New

             10846 - Toaster: [Test Case 1095] Filter doesn't show a dialogue [8]

Other bugs : https://wiki.yoctoproject.org/wiki/WW52_-_2016-12-19_-_Full_Test_Cycle_2.3_M1_rc1#Other_Bugs_found_during_QA_Test_.28NOT_NEW.29

Performance
            Results were only delivered by Ubuntu machine due fedora machine is being used to do debugging activities for bug 10844 [9]
                                                                 M1              M2              %

-        core-image-sato                     01:02:07      01:14:59      +20.71

-        core-image-sato -c rootfs      00:02:47      00:04:14      +52.10

-        core-image-sato (rmwork)     01:00:33      01:08:02      +12.36

-        virtual/kernel                         00:04:53      00:05:01      +2.73

-        eSDK deploy time                  04:23:00       03:47:00     -13.69


pTest

-        Still 12 issues open

-        There were improvements on execution % of busybox, e2fsprogs, openssl

-        There were regression on executions % for lttng-tools, strace, tcl, util-linux


            Full report : https://wiki.yoctoproject.org/wiki/WW05_-_2017-01-24_-_Full_Test_Cycle_2.3_M2_rc1#pTest_for_genericx86-64_on_NUC

Links


1-     http://lists.openembedded.org/pipermail/openembedded-core/2017-January/131986.html

2-     https://bugzilla.yoctoproject.org/show_bug.cgi?id=10964

3-     https://bugzilla.yoctoproject.org/show_bug.cgi?id=10968

4-     https://bugzilla.yoctoproject.org/show_bug.cgi?id=10970

5-     https://bugzilla.yoctoproject.org/show_bug.cgi?id=10983

6-     https://bugzilla.yoctoproject.org/show_bug.cgi?id=10984

7-     https://bugzilla.yoctoproject.org/show_bug.cgi?id=10990

8-     https://bugzilla.yoctoproject.org/show_bug.cgi?id=10846

9-     https://bugzilla.yoctoproject.org/show_bug.cgi?id=10844


Regards,
José

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20170131/aa9cc8c6/attachment.html>