[Automated-testing] Yocto Project Automated Testing Status Update

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Richard Purdie on  Thursday, September 19, 2019 6:24 AM
> 
> I thought a quick summary of where YP is at with automated testing
> maybe interesting for people who don't follow it day to day.

Thanks.  This kind of quick update is very handy to get an idea of
what the different frameworks are doing.

> 
> On a "full" build we now:
> 
> * Run ptests on x86 and arm for ~60 pieces of software
I'm curious about the hardware coverage.  Is this 
done on actual hardware, or in virtual machines?  If it's
on actual hardware, is this in a single location, or distributed
in some different locations?

> * Run LTP and LTP posix tests
> * Test reproducibility of our toolchain and minimal images
> * Run the gcc, binutils and glibc testsuites on all
>   arm/mips/powerpc/x86 32/64 bit
> 
> This all gets summarised in a report like:
> 
> https://autobuilder.yocto.io/pub/non-release/20190916-
> 9/testresults/testresult-report.txt
> 
> which shows we're running over 1.9 million tests on a given full test
> run. We can do this in around 8 hours.
> 
> We're a bit different to most approaches as we're not kernel focused,
> quite the opposite. Ironically the kernel tests are one of the bigger
> holes in our test matrix now.
> 
> Despite that its worth nothing we are finding kernel regressions, I
> wrote a recent LWN article about one (https://lwn.net/Articles/788626/)
> . The report above shows another hinted at with strace ptest's "T" for
> timeout. Its an issue with the 5.2 kernel, we're working on reporting
> it.


That's a very interesting test report.  Thanks!
 -- Tim