[Automated-testing] conventions for test invocation/execution, etc.

[Tim.Bird at sony.com]

> -----Original Message-----
> From: richard.purdie at linuxfoundation.org
> 
> Hi Tim,
> 
> On Thu, 2019-09-05 at 01:40 +0000, Tim.Bird at sony.com wrote:
> > I've been working on some ideas for conventions or standards for test
> > invocation,
> > and I'm not sure how to get a discussion going, so I'm going to just
> > throw out a few
> > ideas and questions, and see where it leads.
> >
> > First, I'm probably mangling this badly, but my understanding is that
> > Yocto Project ptest seeks to split the test invocation into three
> > steps:
> >  - have the build step do a 'make build-test' or something like that
> 
> There are typically two things it needs to do here, compile the tests
> and then secondly do a "make install" of the tests to a specific
> directory. There may be separate targets or a combined one, it depends
> on the software and its build system. Sometimes there is no option but
> to just move files around ourselves.
> 
> >  - provide a per-package custom script that invokes a package's test
> > program
> 
> For each piece of software we install a script as
> /usr/lib/<name>/ptest/run-ptest (we put the ptest files into that
> directory too). Example runners are:
> 
> A nice simple make invocation:
> http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-
> core/libxml/libxml2/run-ptest
> A not so simple execution of test scripts:
> http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-core/util-
> linux/util-linux/run-ptest
> 
> >  - has a test-runner that executes the set of scripts that come with
> > YP/OE packages
> 
> Its a standalone software project you could use independently:
> 
> https://git.yoctoproject.org/cgit.cgi/ptest-runner2/
> 
> It looks for and executes the individual run-ptest scripts and
> processes the output. It finds them as a simple /usr/lib/*/ptest/run-ptest
> search.
> >
> > I heard that someone had some patches for autotools to separate 'make
> > check' or 'make test' into two steps ( a test-build step and a test-
> > execution step).  Is this true?  Can someone provide the details of
> > this to me?  Specifically, what are the  Makefile target names you
> > use for the two different steps, and what are they each supposed to
> > accomplish?
> 
> I think you might be referring to this automake patch we carry:
> 
> http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-
> devtools/automake/automake/buildtest.patch
> 
> It splits "make check" into "make buildtest" and "make runtest". The
> libxml run-ptest script above shows how we'd use runtest on target.
> 
> > I'd like to ask about some additional details.   Does the Yocto
> > Project have a fixed name (or a naming convention) for the script
> > used to execute or wrap the execution of the test functionality for a
> > package?  (eg. something like "runtest.sh", or "<testname>-test.sh")
> 
> Yes, see above.
> 
> Hope that helps a bit!

That's great.  That's exactly the info I was looking for.

Thanks!
 -- Tim