[yocto] Proposed new QA process

[Yeoh, Ee Peng]

Hi Richard, 

Agreed to you inputs! 
I had created a new QA process wiki page to compile all inputs. 
https://wiki.yoctoproject.org/wiki/New_QA_process

> Do we need to add something to the results to indicate which results are from "who"? (i.e. from > the public autobuilder, Intel QA, WR QA, any other sources?). We may want to add something >such as a parameter to the resulttool store command so we can add this info to results?

I had enhanced the resulttool following your suggestion. Patches submitted to openembedded-core mailing list.

Best regards,
Yeoh Ee Peng 

-----Original Message-----
From: richard.purdie at linuxfoundation.org [mailto:richard.purdie at linuxfoundation.org] 
Sent: Wednesday, April 3, 2019 4:18 AM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; 'yocto at yoctoproject.org' <yocto at yoctoproject.org>
Subject: Re: Proposed new QA process

This is a good start. I've filled out some details below and had some questions.

On Tue, 2019-04-02 at 03:28 +0000, Yeoh, Ee Peng wrote:
> Given the new QA tooling (resulttool) available to manage QA test 
> results and reporting, here was the proposed new QA process.
>  
> The new QA process consists below:
> ·       Test Trigger
> ·       Test Planning
> ·       Test Execution
> ·       Test Result Store
> ·       Test Monitoring & Reporting
> ·       Release Decision
>  
> Test Trigger: Each QA team will subscribe to QA notification email 
> (request through Richard Purdie).

The list of notifications is maintained on config.json in the yocto- autobuilder-helper which has a branch per release.
 
> Test Planning: The lead QA team no longer need to setup Testopia and 
> wiki page.  Each QA team (eg. intel, windriver, etc) will perform 
> planning on what extra tests they plan to run and when they'll send 
> the results back, then send these planning information as acknowledge 
> email to QA stakeholders (eg. Richard Purdie, Stephen Jolley) and the 
> lead QA team.  Each QA team can refer to OEQA for automated and manual 
> test cases for their planning.

What form will these notifications take? Is there a time limit for when they'll be received after a QA notification email? Can we agree to include an estimate of execution time in this notification?

> Test Execution: Each QA team will execute the planned extra tests. To 
> make sure test result from the test execution could fully integrated 
> to the new QA tooling (resulttool for test result management and 
> reporting/regression), execute OEQA automated tests and OEQA manual 
> tests through resulttool (refer 
> https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution).
>  
> Test Result Store: Each QA team will store test result to the remote 
> yocto-testresults git repository using resulttool (refer 
> https://wiki.yoctoproject.org/wiki/Resulttool#store), then send the QA 
> completion email (include new defects information) to both QA 
> stakeholder and the lead QA team.  Each QA team will request write 
> access to remote yocto-testresults git repository (request through 
> Richard Purdie).

Ultimately, yes but I want to have things working before we have multiple people pushing things there. Need to document the commands used here to add the results too.

Do we need to add something to the results to indicate which results are from "who"? (i.e. from the public autobuilder, Intel QA, WR QA, any other sources?). We may want to add something such as a parameter to the resulttool store command so we can add this info to results?
 
> Test Monitoring & Reporting: QA stakeholder will monitor testing 
> progress from remote yocto-testresults git repository using resulttool 
> (refer https://wiki.yoctoproject.org/wiki/Resulttool#report).  Once 
> every QA team completed the test execution, the lead QA team will 
> create QA test report and regression using resulttool. Send email 
> report to QA stakeholder and public yocto mailing list.

We should document the command to do this. I'm also wondering where the list of stakeholders would be maintained?

Is Intel volunteering to help with this role for the time being or does someone else need to start doing this.

A key thing we need to document here is that someone, somewhere in this process needs to:

a) Open a bug for each unique QA test failure
b) List the bugs found in the QA report
c) Notice any ptest timeouts and file bugs for those too

> Release Decision: QA stakeholder will make the final decision for 
> release.

The release decision will ultimately be made by the Yocto Project TSC who will review the responses to the QA report (including suggestions from QA) and make a go/nogo decision on that information (exact process to be agreed by the TSC).

Cheers,

Richard


-------------- next part --------------
An embedded message was scrubbed...
From: "Yeoh, Ee Peng" <ee.peng.yeoh at intel.com>
Subject: [PATCH v03] resulttool/merge: enhance merge and control testseries
Date: Thu, 4 Apr 2019 02:30:34 +0000
Size: 2559
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190404/7b5cb29c/attachment-0004.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: "Yeoh, Ee Peng" <ee.peng.yeoh at intel.com>
Subject: [PATCH 2/3] resulttool/store: Enable add EXECUTED_BY config to results
Date: Thu, 4 Apr 2019 02:30:36 +0000
Size: 3428
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190404/7b5cb29c/attachment-0005.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: "Yeoh, Ee Peng" <ee.peng.yeoh at intel.com>
Subject: [PATCH 3/3] resulttool/merge: Enable control TESTSERIES and extra configurations
Date: Thu, 4 Apr 2019 02:30:37 +0000
Size: 4481
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190404/7b5cb29c/attachment-0006.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: "Yeoh, Ee Peng" <ee.peng.yeoh at intel.com>
Subject: [PATCH 1/3] resulttool/resultutils: Enable add extra configurations to results
Date: Thu, 4 Apr 2019 02:30:35 +0000
Size: 4685
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190404/7b5cb29c/attachment-0007.mht>