[Automated-testing] Questions about ptest

[Richard Purdie]

Hi Tim,

Whilst I'm not Tim I can answer some of this.

On Sat, 2019-11-02 at 16:24 +0000, Tim.Bird at sony.com wrote:
> Here's another followup from ATS...
> 
> In my presentation on test definitions, I had some information for
> Yocto Project/OE
> apparently about the wrong thing.
> 
> I thought I was describing ptest, but I think I was referring to an
> on-target oeqa thing.
> I would like to get some information about YP/OE's per-package test
> (or ptests).
> 
> Can you provide me a reference for a ptest that has "characteristic"
> elements?
> Reading through some of the ptest documentation, I see that there is
> apparently
> a ptest option for busybox (that is, the OE core recipe for busybox
> inherits the class ptest).
> 
> What elements of the recipe constitute the ptest test-definition?
> I'm thinking these are things like:
> * RDEPENDS_${PN}-ptest = "zip"
> * do_install_ptest()
> * file://run-ptest (and the script files/run-ptest)
> 
> Has some of the support for ptest been upstreamed into busybox
> itself?
> (for example, the recipe does not contain 'runtest', which is
> referenced by run-ptest.)
> (also, does busybox Makefile already support the required make
> targets for ptest?)
> Maybe there's a patch for adding the required make targets to
> busybox, but if
> so it wasn't obvious to me which patch it was.

The concept of "ptest" in the Yocto Project is that where the source
code for a given project has tests, we package them up into a package
which can be installed onto the target. We do this into a specific
location where we can run them and get the results back in a specific
format.

A given recipe in YP which has ptest support will therefore usually
have:

a) Some additional runtime dependencies for the ptest package

b) Some function to install the tests (like 'make install' but for the
tests)

c) A runner script which converts whatever the tests use into our
execution location and results output format

I made these points correspond to the things you note in the busybox
recipe above.

We talked to upstream busybox and agreed that allowing the tests to be
installed and run on target was a good thing. The "do_install_ptest"
function you referenced above copies a testsuite directory into our
target package. This directory contains 'runtest' which is directly
from busybox upstream.

At this point we can therefore have an image with busybox and by
installing the busybox-ptest package, we can add the tests. These could
be run by anything as we put them in our standard location. We now
build images that contain all our ptests and execute this in our
automated testing.

Busybox is a comparatively nice upstream for this. If you want to see
something more ugly, util-linux is where we just have a function to
pull the bits we need out of the source tree, hack them and then have a
runner to run them on the target.

Another example of an autotools situation is libxml2. There, we use
"make check-am" to compile the tests and patch in a "make install-
ptest" (runtest.patch), we then execute the original project Makefile
on target to run the tests from run-ptest.

Hope that helps, I'll stop here and see if that helps and what further
questions that may raise!

Cheers,

Richard