[yocto] QA cycle report for 2.6 M1 RC1

[Richard Purdie]

Thanks Ee Peng and team, this release seemed to have a number of issues
and proved challenging to QA. I appreciate the work that goes into
these (particularly given the other releases also queued), this is a
useful QA report and has found some real issues.

On Tue, 2018-07-03 at 07:09 +0000, Yeoh, Ee Peng wrote:
> This is the full report for 2.6 M1 RC1: 
> https://wiki.yoctoproject.org/wiki/WW27_-_2018-07-02_-_Full_Test_Cycl
> e_2.6_M1_rc1
>  
> ======= Summary ========
>  
> 99% of planned tests were executed except automated SDK test for
>  (testrun# 9747-9753) and automated SDK image tests
> for core-image-sato-sdk-qemumips (testrun# 9713-9714), where those
> tests were excluded from the automated test suites and it was still
> under investigation why they were excluded.

I think buildgalculator is excluded simply because it takes so long to
run as the mips emulation in qemu is slow. There might be information
in commit logs about why we did that.
 
> There were zero high milestone defect.  Team had found 9 new defects
> where Edgerouter [1] & Beaglebone Black [3] defects were resolved. 
> Team found that pTest for busybox and valgrind had testcases passed
> in previous 2.5 M4 rc1 but failed during this release [8] [9].  Team
> had also found that existing automated xorg test was passed
> successfully while it was not able to catch matchbox window manager
> issue on qemumips [4], thus team had added manual graphic tests as
> temporary solution.  
>  
> ======= QA-Hints========
>  
> Two medium+ defects where 1 was resolved and 1 under design. 
>  
> ======= Bugs ========
>  
> New Bugs
> [1] Bug 12790 - [2.6 M1] Edgerouter can not boot
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12790

[Fixed in master]
 
> [2] Bug 12804 - [QA 2.6 M1 rc1 ][Build Appliance]: proxy issue in VM
> ware
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12804

[Regression where we understand which change broke things but not fixed
yet]
 
> [3] Bug 12795 - [2.6 M1] gcc/g++ doesn't work on beaglebone black
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12795

[Fixed in master]

> [4] Bug 12806 - [2.6 M1 rc1 ][BSP][Test Run 9708]: Qemuppc image was
> not booting with graphic even though runtime/xorg.py test passed
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12806

[Needs further debugging - Ross' comment about arm64 confuses me in a
qemuppc bug :/]
 
> [5] Bug 12832 - [ 2.6 M1 rc1 ][BSP][Test case 267]: audio and video
> does not play in media player[Mturbot x86-64 and NUC7]
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12832

[Not triaged yet]
 
> [6] Bug 12802 - [Yocto-2.6_M1.RC1] Crosstap doesn't work on 2.6 M1
> RC1
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12802

[Unclear if functionality is broken or just a warning message]

> [7] Bug 12813 - [QA 2.6 M1 rc1 ][Toaster]:Build stopped without
> error, to terminate by ctrl+c
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12813

[Valid issue, not release blocker and scheduled for M3]
 
> [8] Bug 12836 - [2.6 M1 RC1] busybox ptest failed
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12836

[Regression: 4 wget tests that passed now fail]

> [9] Bug 12837 - [2.6 M1 RC1] valgrind ptest failed
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12837

[Regression: 19 tests started failing, 2 started passing]


My take on this is that we probably should release M1 rc1 and move to
concentrate on M2 but we need to ensure more of the above issued are
fixed before we can build M2. This QA report tells us very clearly
where we need to focus effort on addressing regressions.

Cheers,

Richard