[yocto] QA cycle report for 2.4.4 RC1

[Jain, Sangeeta]

The test cases run for 2.4.4 RC1 are same as run for 2.4 release. 


>-----Original Message-----
>From: richard.purdie at linuxfoundation.org <richard.purdie at linuxfoundation.org>
>Sent: Saturday, 24 November, 2018 12:07 AM
>To: Jain, Sangeeta <sangeeta.jain at intel.com>; Jolley, Stephen K
><stephen.k.jolley at intel.com>; Eggleton, Paul <paul.eggleton at intel.com>;
>Graydon, Tracy <tracy.graydon at intel.com>; Erway, Tracey M
><tracey.m.erway at intel.com>; yocto at yoctoproject.org
>Cc: Sangal, Apoorv <apoorv.sangal at intel.com>; Kirkiris, Nectar
><nectar.kirkiris at intel.com>
>Subject: Re: QA cycle report for 2.4.4 RC1
>
>On Fri, 2018-11-23 at 03:23 +0000, Jain, Sangeeta wrote:
>>
>> Hello All,
>>
>> This is the full report for 2.4.4 RC1:
>>
>https://wiki.yoctoproject.org/wiki/WW46_-_2018-11-14_-
>_Full_Test_Cycle_2.4.4_RC1
>>
>>
>> Summary
>>
>> All planned tests were executed.
>>
>> Total Test Executed – 3330
>> Passed Test – 3318
>> Failed Test – 8
>> Blocked Test - 4
>>
>> There were zero high priority defect.  Team had found 2 new defects.
>>
>> New Bugs
>>
>> [1] Bug 13033 -  [2.4.4RC1] [Bitbake] [Case 142] PR number is not
>> getting increased with remote server/local client mode
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13033
>>
>>  [2]Bug 13038 - [2.4.4RC1][Package Management][Getting Error Message
>> "Failed to synchronize cache for repo 'repository'"]
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13038
>
>Thanks for running the QA for this. I do have some questions/observations.
>
>Firstly, this report is a little misleading as there are only two bugs mentioned but 8
>failures. The full report shows other bugs which were for example reopened or
>already open.
In report, only new bugs are mentioned. All the bugs which are reopened/already existing are not listed in report mail but can be seen in full report.
However, if its creating any confusion/inconvenience, going forward I can mention all the failing bugs in report.

>
>Taking the above bugs first, I believe they're both manual versions of tests which
>are already automated. I believe the automated tests have passed and there
>appears to be some kind of problem with the manual execution such as the
>wrong process being documented. I'm not quite sure why these are being run
>manually? Was the list of manual tests we received from Intel incorrect?
The list manual test we run for 2.4.4 is same as we run for 2.4 release. This test case is automated later than 2.4 release and is still manual as per 2.4 test plan in Testopia. It's an irony that in Yocto Project we don't have any tracking of a test case converting into automated in which release, and no system that we can update test cases in Dot releases. I am trying to streamline the tracking of test case evolution, so as to avoid such scenarios in future.


>
>Looking at other bugs in the QA report:
>
>https://bugzilla.yoctoproject.org/show_bug.cgi?id=12670
>
>It was reopened as it had been fixed in sumo onwards but not rocko.
>I've backported the fix to the rocko branch. I don't believe its a release blocker.
>
>https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991
>
>This is webkitgtk failing to build in the build-appliance and looks entirely related
>to resources in the VM. How this could have worked in the past yet fail now
>makes me wonder about whether it was in fact tested previously, whether the
>VM configuration changed or quite what happened. I don't believe its a blocking
>bug as it does seem to be a VM resource problem, not a real failure or problem
>with build-appliance.
>Its hard to tell for sure with the limited data we have though.
>
>https://bugzilla.yoctoproject.org/show_bug.cgi?id=12240
>
>A reopened lsb testing bug but its not really showing problems with the actual
>image, just testing difficulties.
>
>
>Based on this, I think 2.4.4 is good to be released, I don't see any blocking issues.
>I would like to understand what tests are actually being run and why though.
>
>Cheers,
>
>Richard
>
Thanks & Regards,
Sangeeta Jain