[Automated-testing] conventions for test invocation/execution, etc.

[richard.purdie at linuxfoundation.org]

Hi Tim,

On Thu, 2019-09-05 at 01:40 +0000, Tim.Bird at sony.com wrote:
> I've been working on some ideas for conventions or standards for test
> invocation,
> and I'm not sure how to get a discussion going, so I'm going to just
> throw out a few
> ideas and questions, and see where it leads.
> 
> First, I'm probably mangling this badly, but my understanding is that
> Yocto Project ptest seeks to split the test invocation into three
> steps:
>  - have the build step do a 'make build-test' or something like that

There are typically two things it needs to do here, compile the tests
and then secondly do a "make install" of the tests to a specific
directory. There may be separate targets or a combined one, it depends
on the software and its build system. Sometimes there is no option but
to just move files around ourselves.

>  - provide a per-package custom script that invokes a package's test
> program

For each piece of software we install a script as
/usr/lib/<name>/ptest/run-ptest (we put the ptest files into that
directory too). Example runners are:

A nice simple make invocation:
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-core/libxml/libxml2/run-ptest
A not so simple execution of test scripts:
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-core/util-linux/util-linux/run-ptest

>  - has a test-runner that executes the set of scripts that come with
> YP/OE packages

Its a standalone software project you could use independently:

https://git.yoctoproject.org/cgit.cgi/ptest-runner2/

It looks for and executes the individual run-ptest scripts and
processes the output. It finds them as a simple /usr/lib/*/ptest/run-ptest search.
> 
> I heard that someone had some patches for autotools to separate 'make
> check' or 'make test' into two steps ( a test-build step and a test-
> execution step).  Is this true?  Can someone provide the details of
> this to me?  Specifically, what are the  Makefile target names you
> use for the two different steps, and what are they each supposed to
> accomplish?

I think you might be referring to this automake patch we carry:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/automake/automake/buildtest.patch

It splits "make check" into "make buildtest" and "make runtest". The
libxml run-ptest script above shows how we'd use runtest on target.

> I'd like to ask about some additional details.   Does the Yocto
> Project have a fixed name (or a naming convention) for the script
> used to execute or wrap the execution of the test functionality for a
> package?  (eg. something like "runtest.sh", or "<testname>-test.sh")

Yes, see above.

Hope that helps a bit!

Cheers,

Richard