[yocto] QA Test Report for Yocto Project Release 2.3 M3 rc1

Thu Mar 23 13:03:25 PDT 2017

Hi

Here is the report for Full  QA Cycle on Release 2.3 M3 rc1

Full Report : https://wiki.yoctoproject.org/wiki/WW11_-_2017-03-15_-_Full_Test_Cycle_2.3_M3_rc1

Summary

The QA cycle for point release 2.3 M3 rc1 .1 is complete. There are 12 new issues found, 4 of them are High and 5 are M+. 2 of the high were blocking Performance and Selftest execution and were quickly fixed and merged, hence those 2 test suites were executed in a different commit from the one tagged for M3 rc1. In total there are 20 bugs attached to 2.3 M3 rc1, 5 High, 7 M+ and 8 M. Also there are still 15 M+ to be implemented still tagged to 2.3 M3 release.  Regarding to Performance there is huge improvements on rootfs measurements on both machines Fedora and Ubuntu and eSDK on Ubuntu, and some little regressions (around 3%) on both machines in different tasks. pTest shows improvements on acl and e2fsprogs moudles  but also shows regression on other modules.

QA Hints

-        There are still may high and M+ bugs that are opened against M3 rc1

-        2 of the highest test suites (Selftest and Performance) were not executed against M3 rc1 commit, even when they did show up good results on the commit were tested on the commit tagged for 2.3 M3 rc1 are failing.

-        There are still 15 M+ tagged to be implemented on 2.3 M3, hence a decision should be taken about them, implemented for M4 or moved to 2.4

Based on above points the strategy to follow up form QA perspective is to have another release candidate when the blocker issues get solved and hence have a better quality on M3 and avoid High issues on M4.

=== Details ===
Bugs

      * New

-        High

o   11154 - buildperf: Fedora machine failing on buildperf execution due could not find 'ip' tool [1]

o   11172 - bitbake: Fetcher failure when building core-image-minimal with "ERROR: socat: not found" [2]

o   11193 - runqemu cannot launch image if enable "rm_work" [3]

o   11223 - devtool runqemu doesn't work in eSDK [4]

-        M+

o   11224 - Compliance: LTP, POSIX, LSB tests are not creating results file (.targz) [5]

o   11222 - [eclipse] C compiler cannot create executables when reconfiguring project [6]

o   11174 - Update yocto-bsp script for kernel 4.10 [7]

o   11144 - Build Appliance : Toaster requires pod2man [8]

o   11185 - selftest/devtool: failing when parsing "mraa" recipe [9]

-        M / Low

o   11173           testimage: "test_dnf_install_from_http" is failing [10]

o   11220           runqemu: Qemux86 not boot with Ubuntu [11]

o   11225           Build from Command line fail when finish all the tasks [12]

      * High / M+ Not New

-        11119           runqemu-extract-sdk is failing when extracting .tar.bz file [13]

-        11064           [Yocto 2.2.1] kernel version mismatched when using yocto-bsp to create custom qemu bsp [14]

-        11025           selftest/devtool: Build mdadm failing on Centos7 and Ubuntu 1404 [15]

Full bugs Report : https://wiki.yoctoproject.org/wiki/WW11_-_2017-03-15_-_Full_Test_Cycle_2.3_M3_rc1#Bugs_Found_during_QA_Test

Performance
Performance measurements were huge  improved  for rootfs on both machines (Ubuntu around 30%, Fedora 20%)  falling down the numbers  to the average observed before the "recipe specific sysroot" implementation. On Ubuntu there was also a very good improvement  on eSDK deploy time of around 12%, some other measurement times were up on around 3% on both machines, below are the tables with the times of the tests:

-        Ubuntu

Test

2.3 M2 rc3

2.3 M3 rc1

%

sato

1:11:47

1:10:15

-2.14

rootfs

4:02

2:28

-38.84

rmwork

1:05:56

1:05:28

-0.71

kernel

5:00

5:11

+3.67

eSDK

3:28

3:01

-12.98

-        Fedora

Test

2.3 M2 rc3

2.3 M3 rc1

%

sato

1:12:58

1:12:34

-0.55

rootfs

3:19

2:40

-19.60

rmwork

1:04:50

1:07:10

+3.60

kernel

6:27

6:29

+0.52

eSDK

3:11

3:17

+3.14

Performance Charts : https://wiki.yoctoproject.org/charts/perf_milestone_GDC/performance_test.html

pTest

-        There were improvemnte of pass rate for acl and e2fsprogs modules

-        Modules that decreased pass rate were glib-2.0, openssh, openssl, quilt, strace, valgrind

-        A new bug was raised to address quil failures [15]

pTest full report : https://wiki.yoctoproject.org/wiki/WW11_-_2017-03-15_-_Full_Test_Cycle_2.3_M3_rc1#pTest_for_genericx86-64_on_NUC

Kernel

-        to test case "TCTEMP_2.3_MANUAL_Kernel_dev_External_source" was run, however due an issue with git, currently there is no way to test with 100% of reliability. I also tested on different machines. An empty directory is released when you are trying to clone from remote and local. I  tested it also with older kernel 4.4, 4.6 and I got the same issue. One month ago I could reproduce this properly.

Links

1.      https://bugzilla.yoctoproject.org/show_bug.cgi?id=11154

2.      https://bugzilla.yoctoproject.org/show_bug.cgi?id=11172

3.      https://bugzilla.yoctoproject.org/show_bug.cgi?id=11193

4.      https://bugzilla.yoctoproject.org/show_bug.cgi?id=11223

5.      https://bugzilla.yoctoproject.org/show_bug.cgi?id=11224

6.      https://bugzilla.yoctoproject.org/show_bug.cgi?id=11222

7.      https://bugzilla.yoctoproject.org/show_bug.cgi?id=11174

8.      https://bugzilla.yoctoproject.org/show_bug.cgi?id=11144

9.      https://bugzilla.yoctoproject.org/show_bug.cgi?id=11185

10.   https://bugzilla.yoctoproject.org/show_bug.cgi?id=11173

11.   https://bugzilla.yoctoproject.org/show_bug.cgi?id=11220

12.   https://bugzilla.yoctoproject.org/show_bug.cgi?id=11225

13.   https://bugzilla.yoctoproject.org/show_bug.cgi?id=11119

14.   https://bugzilla.yoctoproject.org/show_bug.cgi?id=11064

15.   https://bugzilla.yoctoproject.org/show_bug.cgi?id=11025

16.   https://bugzilla.yoctoproject.org/show_bug.cgi?id=11221

Regards,
José

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20170323/a8069dd9/attachment.html>