[Automated-testing] conventions for test invocation/execution, etc.

[Tim.Bird at sony.com]

Hello everyone,

I've been working on some ideas for conventions or standards for test invocation,
and I'm not sure how to get a discussion going, so I'm going to just throw out a few
ideas and questions, and see where it leads.

First, I'm probably mangling this badly, but my understanding is that Yocto Project
ptest seeks to split the test invocation into three steps:
 - have the build step do a 'make build-test' or something like that
 - provide a per-package custom script that invokes a package's test program
 - has a test-runner that executes the set of scripts that come with YP/OE packages

I heard that someone had some patches for autotools to separate 'make check' or
'make test' into two steps ( a test-build step and a test-execution step).  Is this
true?  Can someone provide the details of this to me?  Specifically, what are the 
Makefile target names you use for the two different steps, and what are they
each supposed to accomplish?

I'd like to ask about some additional details.   Does the Yocto Project have a fixed
name (or a naming convention) for the script used to execute or wrap the execution
of the test functionality for a package?  (eg. something like "runtest.sh", or "<testname>-test.sh")

If I recall correctly, doesn't CKI support a specific name for the test program: runtest.sh?

Do any other systems do this (use a specific name)?

Fuego uses the directory name for the test, with a shell script called "fuego_test.sh" for the
host-side script. Currently Fuego has no standard for the name for the DUT-side wrapper script.
However, many of our tests use the name "<testname>_test.sh".

Please let me know if your test framework uses a naming convention or standard for the
test program or test program wrapper script in your system.

Thanks,
 -- Tim