In the 3.0 Zeus release, we have added ‘resulttool’ and ‘yocto-testresults’. This new age of enlightenment gives us much more data about how the builds are proceeding and a measure of the health of each of the components. What is resulttool? How do we interpret the results? What can we do with those results?
Embrace the power of the Force and realize that we can now perform Anomaly Detection on the results to catch regressions that a Human needs to pay attention to. Do humans need to look at every TestRun and every TestResult? NO. One thousand times NO. With some automation, we can ingest the testresults and perform Anomaly Detection on the results. We can then flag egregious offenders to the historic results and highlight the differences. This allows us to repurpose high-value QA resources to investigate regressions and unexpected results, rather than manually testing or manually investigating test results. In this presentation, we discuss some pre-release approaches to ingesting the test cases and test results. We also discuss some approaches to algorithmically detect significant changes (regressions) in the results over time.
Speaker: Tim Orling, Sr. Linux Software Engineer, Intel Corporation