[yocto] QA cycle report for 2.4 M2 rc3

[Cruz, Libertad]

Hello All,



Here is the report for the 2.4 M2 rc3 full release test cycle.

Full Report :  https://wiki.yoctoproject.org/wiki/WW32_-_2017-08-09_-_Full_Test_Cycle_2.4_M2_rc3



======= Summary ========

The QA cycle for release 2.4 M2 rc3 is complete.  There are two new bugs, 11923[2] medium + impacts 28.6% of eclipse test run and 11917[1] medium + is not a critical blocker since it only impacts 0.8% of oe-selftest on Fedora 24. The one bug that worries QA is 11923[2]  which impacts the eclipse testcases responsible for verifying the process of project debugging, workaround for this can be that user uses eclipse-plugin in 2.4 M2 rc2 which are stable.



On the wiki a  new section was created for Compliance results. There is currently an issue with the hardware infrastructure of Wind River hence  beaglebone and edgerouter have not been fully tested.


Performance
There were no relevant changes on measurements compared to  2.4 M1 rc2.



Ubuntu                Test        2.4 M2 rc2          2.4 M2 rc3          %

                                sato       1:13:58                             1:14:08             0.23

                                rootfs    2:31                    2:31                   0.00

                                rmwork                1:07:22                             1:06:40           -1.04

                                kernel    5:39                    5:32                  -2.06

                                eSDK      2:30                    2:32                  1.33





Fedora                  Test        2.4 M2 rc2          2.4 M2 rc3          %

                                sato       1:13:52                             1:12:25           -1.96

                                rootfs    2:29                    2:25                  -2.68

                                rmwork                1:08:42                             1:07:48           -1.31

                                kernel    6:04                    5:58                  -1.65

                                eSDK      2:34                    2:34                  0.00





ptest

There were no relevant changes on ptest compared to 2.3 M2 rc2



Compliance

For posix test there is no changes in the number of failures when comparing  2.4m2 rc2 and 2.4 m2 rc3.



                    Failures            2.4 M2 RC3      2.4 M1 RC2

                  Generic86-64           46                      46

                  Beaglebone              45                      45

                  Edgerouter               46                      46

                  Mpc8315e-rdb        45                      45



Results are found here: https://wiki.yoctoproject.org/wiki/Posix_result

Detail analysis found here:  https://wiki.yoctoproject.org/wiki/POSIX-results



On 2.4 M2 rc3  LTP testing was also completed for beaglebone and we had an less test failed on Genericx86-64 and mpc8315e-rdb when comparing results with 2.4 M1 rc 2.



                                                                Genericx86-64     beaglebone       mpc8315e-rdb

2.4 M2 rc3                Total                        1384                          1346                    1346

                                     Failures                    23                               15                          27



                                                                Genericx86-64                                    mpc8315e-rdb

2.4 M1 rc 2               Total                        1384                                                       1348

                                     Failures                    24                                                            30





The LPT results are found: https://wiki.yoctoproject.org/wiki/LTP_result .





======= QA-Hints========



QA has found 2.4 m2 rc3  to be stable and all commits provided as workarounds in 2.4 m2 rc2 were successfully integrated. QA approves to this build candidate being released.





======= Bugs ========



       New Bugs
            -11917[1]  (buildoptions) Failure on test_arch_work_dir and layer_with_out_git_dir
            -11923[2] [Eclipse] Failed to debug a Project in Eclipse and displays error: java.lang.NullPointerException
       M+ bugs
            -11879[3]  Testimage: multiple "uvesafb" errors present in log in minnowmax on genericx86/genericx86-64
            -11833[4] Toaster timeout for bitbake server






Full Bug Report : https://wiki.yoctoproject.org/wiki/WW32_-_2017-08-09_-_Full_Test_Cycle_2.4_M2_rc3#Bugs_Found_during_QA_Test





======== Links =========



    1.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11917 [1]

    2.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11923 <https://bugzilla.yoctoproject.org/show_bug.cgi?id=11923%20> [2]

    3.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11879 <https://bugzilla.yoctoproject.org/show_bug.cgi?id=10832%20> [3]

    4.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11833 <https://bugzilla.yoctoproject.org/show_bug.cgi?id=11833%20> [4]





Regards

Libertad G.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20170810/f87cedb3/attachment.html>