[yocto] QA cycle report for 2.5.3 RC3

[Jain, Sangeeta]

>-----Original Message-----
>From: richard.purdie at linuxfoundation.org <richard.purdie at linuxfoundation.org>
>Sent: Tuesday, 23 April, 2019 10:31 PM
>To: Jain, Sangeeta <sangeeta.jain at intel.com>; 'yocto at yoctoproject.org'
><yocto at yoctoproject.org>; Eggleton, Paul <paul.eggleton at intel.com>; 'Michael
>Halstead' <mhalstead at linuxfoundation.org>; Erway, Tracey M
><tracey.m.erway at intel.com>; 'sjolley.yp.pm at gmail.com'
><sjolley.yp.pm at gmail.com>; 'openembedded-core-
>request at lists.openembedded.org' <openembedded-core-
>request at lists.openembedded.org>
>Cc: Sangal, Apoorv <apoorv.sangal at intel.com>; Yeoh, Ee Peng
><ee.peng.yeoh at intel.com>; Graydon, Tracy <tracy.graydon at intel.com>;
>Tummalapalli, Vineela <vineela.tummalapalli at intel.com>
>Subject: Re: QA cycle report for 2.5.3 RC3
>
>On Thu, 2019-04-18 at 04:58 +0000, Jain, Sangeeta wrote:
>>
>> QA cycle report for 2.5.3 RC3:
>>
>> No high milestone defects.
>> Test results are available at following location:
>> ·        For results of all automated tests, refer to results at
>> public AB [1].
>> ·        For other test results, refer to attachment [2].
>> ·        For test report for test cases run by Intel and WR team,
>> refer attachment [3]
>> ·        For full test report, refer attachment [4]
>> ·        For ptest results, please refer to results at public AB [5]
>> ·        For ptest report, refer to attachment [6]
>> No new defects are found in this cycle.
>> Number of existing issues observed in this release is 2- toaster [7]
>> and Build-appliance [8] For ptest, regression data is not available
>> for this release. No timeout issues.
>> Test result report on Public AB shows no failures.
>
>I've been discussing with Tracy and Vineela how to best handle the test results for
>the release for 2.5.3. In the end I:
>
>a) Copied in the ptest results to the right place. This will happen automagically in
>all future builds:
>
>https://autobuilder.yocto.io/pub/releases/yocto-2.5.3/testresults/qemux86-64-
>ptest/

>
>b) Copied in the intel test results to their own directory:
>
>https://autobuilder.yocto.io/pub/releases/yocto-2.5.3/testresults-intel/
>
>c) Manually generated an updated test report for the combined results with the
>commands:
>
>$ cd /srv/autobuilder/autobuilder.yoctoproject.org/pub/releases/yocto-2.5.3
>$resulttool report . > testreport.txt
>
>d) Added a header to the report which consisted of the report from QA with
>details of the bugs etc.
>
>This means we have a top level testreport file which contains all the test
>information about the release.
>
>I'd like to make this the standard procedure for release. The release notes and
>announcement can refer to the test report included with the release and the test
>results as its there all together.
>
>Ultimately I'd like to improve the formatting of the report (separate out ptest
>regressions from the other regressions, maybe html, maybe graphs, better
>regression information) but that is something for the future.
>
>Does that work for everyone?

That looks great to me. 
As a further improvement for sharing of test results, I am working to create a wiki page for consolidated 
test results. For 2.5.3, it can be viewed at:
https://wiki.yoctoproject.org/wiki/WW16_-_2019-04-18_-_Full_Test_Cycle_2.5.3_RC3

After you have added more results on AB, I will change some existing links on this web page with the new ones.

Another thought I have is that since for any release, all data will be removed from public AB after few days of release,
Should I include here a  link to "yocto-testresults" git repo? It will make this wiki page more useful for reference in future as well.



>
>Cheers,
>
>Richard
>
Thanks & Regards,
Sangeeta Jain