[Automated-testing] Yocto Project Automated Testing Status Update

[Richard Purdie]

I thought a quick summary of where YP is at with automated testing
maybe interesting for people who don't follow it day to day.

On a "full" build we now:

* Run ptests on x86 and arm for ~60 pieces of software
* Run LTP and LTP posix tests
* Test reproducibility of our toolchain and minimal images
* Run the gcc, binutils and glibc testsuites on all 
  arm/mips/powerpc/x86 32/64 bit

This all gets summarised in a report like:

https://autobuilder.yocto.io/pub/non-release/20190916-9/testresults/testresult-report.txt

which shows we're running over 1.9 million tests on a given full test
run. We can do this in around 8 hours.

We're a bit different to most approaches as we're not kernel focused,
quite the opposite. Ironically the kernel tests are one of the bigger
holes in our test matrix now.

Despite that its worth nothing we are finding kernel regressions, I
wrote a recent LWN article about one (https://lwn.net/Articles/788626/)
. The report above shows another hinted at with strace ptest's "T" for
timeout. Its an issue with the 5.2 kernel, we're working on reporting
it.

Cheers,

Richard