[yocto] QA cycle report for 2.3.3 RC1

[Cruz, Libertad]

Hello All,
Enjoy viewing the full Report for 2.3.3 RC1:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1

======= Summary ========

The QA cycle for release 2.3.3 RC1 is complete.  There are 4 new bugs from which so far none of them are high. QA has two big concerns:

        1.  Performance report shows an increase of 30% build time on the eSDK in fedora 23, bug has not been created until further investigation with setup outside of the GDC environment. 
        2.  ptest comparison script is giving wrong results, bug 12441 was opened. 


-----

ptest

Package slang show 0% pass rate in 2.3.3 rc1, the results need to be validated, because bug 12441 impacts the comparison script.

-----

Toaster
Exploratory testing was executed in this release.

-----
Performance

The performance results showed regression on some of the measurements compared to 2.3.2rc1 info showed below, the most important one is on building eSDK that is taking around 30% of increase on fedora machine.


Ubuntu 15                          Test       2.3.2 rc1              2.3.3 rc1              %
                                             sato       1:12:11                1:16:29                +5.96
                                             rootfs    2:49                      2:53                      +2.37
                                             rmwork 1:04:15                1:06:47                +3.94
                                             kernel    5:21                      5:25                      +1.25
                                             eSDK      2:46                      2:46                      0.00
                                                                           
                                                                           
Fedora 23                           Test       2.3.2 rc1              2.3.3 rc1              %
                                             sato       1:14:51                1:21:20                +8.66
                                             rootfs    2:45                      2:44                      -0.61
                                             rmwork 1:07:47                1:14:54                +10.50
                                             kernel    5:56                      6:04                      +2.25
                                             eSDK      3:06                      4:12                      +35.48


https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1#Performance_Charts

======= QA-Hints========


The build is stable enough to be released, provided the following results are checked and investigated:

    1. Performance needs to be executed in an environment outside of GDC in another fedora 23, volunteers?
    2. QA asks ptest for slang package be checked by development to verify that it is executing.

======= Bugs ========

       New Bugs
            -12440[1] test_qemu_efi (oeqa.selftest.wic.Wic) fails on Fedora26
            -12441[2] ptest comparison script is not showing correct results
            -12436[3] [Test Case 1958] Project names cannot be blank
            -12437[4] [Test Case 1958] Toaster exploratory testingin the test run.

       Not new M+ bugs
            -12022[5] signing.Signing.test_signing_packages hangs on ubuntu1704.yocto.io

Full Bug Report:  https://wiki.yoctoproject.org/wiki/WW51_-_2017-12-20-_Full_Test_Cycle_-_2.3.3_rc1#Bugs_Found_during_QA_Test



======== Links =========
    1.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12440[1] 

    2.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12441[2] 

    3.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12436[3] 

    4.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12437[4] 

    5.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12022 [5] 
   

Regards
Libertad G.