[yocto] QA cycle report for 2.7 M3 RC1

[Jain, Sangeeta]

Hi Richard,

For oe-selftest -r runqemu,meta_ide, configuration we are using is only changing the machine type.

For adding "who" identifier in resulttool command, we can take it up as an enhancement.

Thanks & Regards,
Sangeeta Jain

>-----Original Message-----
>From: richard.purdie at linuxfoundation.org <richard.purdie at linuxfoundation.org>
>Sent: Wednesday, 3 April, 2019 2:32 AM
>To: Jain, Sangeeta <sangeeta.jain at intel.com>; 'yocto at yoctoproject.org'
><yocto at yoctoproject.org>; Eggleton, Paul <paul.eggleton at intel.com>; 'Michael
>Halstead' <mhalstead at linuxfoundation.org>; Erway, Tracey M
><tracey.m.erway at intel.com>; sjolley.yp.pm at gmail.com; Burton, Ross
><ross.burton at intel.com>
>Cc: OTC Embedded Malaysia <otc.embedded.malaysia at intel.com>
>Subject: Re: QA cycle report for 2.7 M3 RC1
>
>Hi Sangeeta/Ee Peng,
>
>On Tue, 2019-04-02 at 05:02 +0000, Jain, Sangeeta wrote:
>> QA cycle report for 2.7 M3 RC1:
>>
>> No high milestone defects.
>> This is completion of first QA cycle after adoption of new QA process.
>> Test results are available at following location:
>> ·        For results of all automated tests, please refer to results
>> at public AB [1].
>> ·        For other test results, refer to attachment [2].
>> ·        For test report for test cases run by Intel and WR team,
>> refer attachment [3]
>> ·        For full test report, refer attachment [4]
>> 2 new defects are found in this cycle, oe-selftest [5] and qemu-
>> shutdown[6].
>> Number of existing issues observed in this release is 3- toaster [7],
>> Build-appliance [8] and Beaglebone [9] For ptest, 3 package tests are
>> failing in this release which were passing in previous release: python
>> [10], python3[11] and strace[12].
>> 3 packages are facing timeout issues: lttng-tools [13], openssh [14]
>> and python3 [15].
>
>Thanks for this. I have some questions/observations.
>
>Looking at the Intel/WR report, I see tests which start oeselftest_XXX for sdk-
>extras and qemu-shutdown. I think this means you have some QA test
>infrastructure running "oe-selftest -r runqemu,meta_ide", iterating over
>MACHINE=qemuppc, qemumips, qemuarm, qemux86?
>
>Is this some automation we're missing in the autobuilder configuration?
>We all other tests covered and this is the only remaining automated piece we
>haven't covered? What configuration are we missing exactly for the missing
>tests?

Just run for MACHINE=qemuppc /qemumips / qemuarm / qemux86

>
>I'm partly wondering how you found the shutdown issue but this would explain
>that.
>
>I think going forward we need to add some kind of "who" identifier to the test
>results so we can say where/who they came from (main autobuilder, Intel, WR,
>anywhere else). We could add a resulttool command to "stamp" a set of test
>results with this, maybe as a parameter to the store command?

Will work on resulttool to add this enhancement. Though not sure about timeline as of now.

>
>Of the bugs, I didn't see any which I think should block M3, I think we probably
>should release that and move forward to M4. I do want to see some of these
>fixed before we build M4 though. I've made comments on them below.
>
>> New Bugs :
>>
>> [5] Bug 13237 - [2.7 M3 RC1][OE-selftest][Opensuseleap 15.0][Public
>> AB]:"test_wic_cp_ext" failing for Opensuseleap 15.0.
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13237
>
>Ross: Are you looking into that one?
>
>> [6] Bug 13234 - [2.7 M3 RC1] qemumips & qemumips64: failed to shutdown
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13234
>
>This needs looking into and fixing, as well as figuring out why the autobuilder
>doesn't see this (see above for my hypothesis).
>
>> Previous Bugs observed in this release:
>>
>> [7] Bug 13191 – [Test Case 1439] Build Failure after adding or
>> removing packages with and without dependencies
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13191
>>
>> [8] Bug 12991 - [Bug] [2.6 M4 RC1][Build-Appliance] Bitbake build-
>> appliance-image getting failed during building image due to webkitgtk
>> package.
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991
>
>We need to do something about this. I believe its related to resource starvation in
>the VM.
>
>> [9] Bug 13153 – [2.7 M2 RC1] Systemtap doesn't work on beaglebone
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13153
>
>I'm hoping a SRCREV bump for beaglebone-yocto fixes this.
>
>> ptest Bugs:
>>
>> [10] Bug 13249 - [2.7 M3 RC1] python ptest failure
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13249
>
>I'm hoping a patch from Ross in master fixes this, need to retest ptest with this
>applied.
>
>> [11] Bug 13253 - [2.7 M3 RC1] python3 ptest failure
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13253
>
>I'm hoping a patch from Ross in master fixes this, need to retest ptest with this
>applied.
>
>> [12] Bug 13254 - [2.7 M3 RC1] strace ptest failure
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13254
>
>No ideas on this one yet.
>
>> [13] Bug 13255 - [2.7 M3 rc1] lttng-tools ptest facing timeout issue
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13255
>
>There are patches which are due to arrive soon which should address this.
>
>> [14] Bug 13256 - [2.7 M3 rc1] openssh ptest facing timeout issue
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13256
>
>No ideas on this one yet.
>
>> [15] Bug 13257 - [2.7 M3 rc1] python3 ptest facing timeout issue
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13257
>
>I'm hoping a patch from Ross in master fixes this, need to retest ptest with this
>applied.
>
>Cheers,
>
>Richard