[yocto] Proposed new QA process

[richard.purdie at linuxfoundation.org]

This is a good start. I've filled out some details below and had some
questions.

On Tue, 2019-04-02 at 03:28 +0000, Yeoh, Ee Peng wrote:
> Given the new QA tooling (resulttool) available to manage QA test
> results and reporting, here was the proposed new QA process.
>  
> The new QA process consists below:
> ·       Test Trigger
> ·       Test Planning
> ·       Test Execution
> ·       Test Result Store
> ·       Test Monitoring & Reporting
> ·       Release Decision
>  
> Test Trigger: Each QA team will subscribe to QA notification email
> (request through Richard Purdie).

The list of notifications is maintained on config.json in the yocto-
autobuilder-helper which has a branch per release.
 
> Test Planning: The lead QA team no longer need to setup Testopia and
> wiki page.  Each QA team (eg. intel, windriver, etc) will perform
> planning on what extra tests they plan to run and when they'll send
> the results back, then send these planning information as acknowledge
> email to QA stakeholders (eg. Richard Purdie, Stephen Jolley) and the
> lead QA team.  Each QA team can refer to OEQA for automated and
> manual test cases for their planning.

What form will these notifications take? Is there a time limit for when
they'll be received after a QA notification email? Can we agree to
include an estimate of execution time in this notification?

> Test Execution: Each QA team will execute the planned extra tests. To
> make sure test result from the test execution could fully integrated
> to the new QA tooling (resulttool for test result management and
> reporting/regression), execute OEQA automated tests and OEQA manual
> tests through resulttool (refer 
> https://wiki.yoctoproject.org/wiki/Resulttool#manualexecution).
>  
> Test Result Store: Each QA team will store test result to the remote
> yocto-testresults git repository using resulttool (refer 
> https://wiki.yoctoproject.org/wiki/Resulttool#store), then send the
> QA completion email (include new defects information) to both QA
> stakeholder and the lead QA team.  Each QA team will request write
> access to remote yocto-testresults git repository (request through
> Richard Purdie).

Ultimately, yes but I want to have things working before we have
multiple people pushing things there. Need to document the commands
used here to add the results too.

Do we need to add something to the results to indicate which results
are from "who"? (i.e. from the public autobuilder, Intel QA, WR QA, any
other sources?). We may want to add something such as a parameter to
the resulttool store command so we can add this info to results?
 
> Test Monitoring & Reporting: QA stakeholder will monitor testing
> progress from remote yocto-testresults git repository using
> resulttool (refer 
> https://wiki.yoctoproject.org/wiki/Resulttool#report).  Once every QA
> team completed the test execution, the lead QA team will create QA
> test report and regression using resulttool. Send email report to QA
> stakeholder and public yocto mailing list.

We should document the command to do this. I'm also wondering where the
list of stakeholders would be maintained?

Is Intel volunteering to help with this role for the time being or does
someone else need to start doing this.

A key thing we need to document here is that someone, somewhere in this
process needs to:

a) Open a bug for each unique QA test failure
b) List the bugs found in the QA report
c) Notice any ptest timeouts and file bugs for those too

> Release Decision: QA stakeholder will make the final decision for
> release.

The release decision will ultimately be made by the Yocto Project TSC
who will review the responses to the QA report (including suggestions
from QA) and make a go/nogo decision on that information (exact process
to be agreed by the TSC).

Cheers,

Richard