[Automated-testing] conventions for test invocation/execution, etc.

[Milosz Wasilewski]

On Thu, 5 Sep 2019 at 02:43, <Tim.Bird at sony.com> wrote:
>
> Hello everyone,
>
> I've been working on some ideas for conventions or standards for test invocation,
> and I'm not sure how to get a discussion going, so I'm going to just throw out a few
> ideas and questions, and see where it leads.
>
> First, I'm probably mangling this badly, but my understanding is that Yocto Project
> ptest seeks to split the test invocation into three steps:
>  - have the build step do a 'make build-test' or something like that
>  - provide a per-package custom script that invokes a package's test program
>  - has a test-runner that executes the set of scripts that come with YP/OE packages
>
> I heard that someone had some patches for autotools to separate 'make check' or
> 'make test' into two steps ( a test-build step and a test-execution step).  Is this
> true?  Can someone provide the details of this to me?  Specifically, what are the
> Makefile target names you use for the two different steps, and what are they
> each supposed to accomplish?
>
> I'd like to ask about some additional details.   Does the Yocto Project have a fixed
> name (or a naming convention) for the script used to execute or wrap the execution
> of the test functionality for a package?  (eg. something like "runtest.sh", or "<testname>-test.sh")
>
> If I recall correctly, doesn't CKI support a specific name for the test program: runtest.sh?
>
> Do any other systems do this (use a specific name)?

In test-definitions we try to follow LAVA convention. So there is no
specific script naming convention however we try to use <testname>.sh.
Sometimes there are more than one script to execute the test. There is
additional YAML test description that can be consumed by LAVA to
execute tests. This way tests can be executed in LAVA, standalone
using script and using test-runner.py from test-definitons. There is
nothing that prevents us from renaming all scripts to say
'runtest.sh'. It's just not very high on the priority list.

milosz

>
> Fuego uses the directory name for the test, with a shell script called "fuego_test.sh" for the
> host-side script. Currently Fuego has no standard for the name for the DUT-side wrapper script.
> However, many of our tests use the name "<testname>_test.sh".
>
> Please let me know if your test framework uses a naming convention or standard for the
> test program or test program wrapper script in your system.
>
> Thanks,
>  -- Tim
>
> --
> _______________________________________________
> automated-testing mailing list
> automated-testing at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/automated-testing