[Automated-testing] conventions for test invocation/execution, etc.

[Veronika Kabatova]

----- Original Message -----
> From: "Tim Bird" <Tim.Bird at sony.com>
> To: automated-testing at yoctoproject.org
> Cc: "richard purdie" <richard.purdie at linuxfoundation.org>, vkabatov at redhat.com
> Sent: Thursday, September 5, 2019 3:40:59 AM
> Subject: conventions for test invocation/execution, etc.
> 
> Hello everyone,
> 
> I've been working on some ideas for conventions or standards for test
> invocation,
> and I'm not sure how to get a discussion going, so I'm going to just throw
> out a few
> ideas and questions, and see where it leads.
> 
> First, I'm probably mangling this badly, but my understanding is that Yocto
> Project
> ptest seeks to split the test invocation into three steps:
>  - have the build step do a 'make build-test' or something like that
>  - provide a per-package custom script that invokes a package's test program
>  - has a test-runner that executes the set of scripts that come with YP/OE
>  packages
> 
> I heard that someone had some patches for autotools to separate 'make check'
> or
> 'make test' into two steps ( a test-build step and a test-execution step).
> Is this
> true?  Can someone provide the details of this to me?  Specifically, what are
> the
> Makefile target names you use for the two different steps, and what are they
> each supposed to accomplish?
> 
> I'd like to ask about some additional details.   Does the Yocto Project have
> a fixed
> name (or a naming convention) for the script used to execute or wrap the
> execution
> of the test functionality for a package?  (eg. something like "runtest.sh",
> or "<testname>-test.sh")
> 
> If I recall correctly, doesn't CKI support a specific name for the test
> program: runtest.sh?
> 

CKI uses Beaker [0] so we are forced to use the invocation Beaker uses. This
means having a Makefile with a "make run" target (which can call build if
needed) that executes the test. The test file is called runtest.sh as you
mentioned. I don't think this naming is enforced by Beaker and you should be
able to put a different file to be executed into the Makefile if you want to
but I've never seen a different name used in practice so far.

[0] https://beaker-project.org


Veronika

> Do any other systems do this (use a specific name)?
> 
> Fuego uses the directory name for the test, with a shell script called
> "fuego_test.sh" for the
> host-side script. Currently Fuego has no standard for the name for the
> DUT-side wrapper script.
> However, many of our tests use the name "<testname>_test.sh".
> 
> Please let me know if your test framework uses a naming convention or
> standard for the
> test program or test program wrapper script in your system.
> 
> Thanks,
>  -- Tim
> 
>