[Automated-testing] Test definition survey

[Milosz Wasilewski]

On Fri, 11 Jan 2019 at 01:41, <Tim.Bird at sony.com> wrote:
>
> OK everyone.  My holidays are over, and it's time to roll up my sleeves and get back to work
> on automated testing standards.
>
> One of my action items from the Automated Testing Summit
> (see https://elinux.org/Automated_Testing_Summit)
> was to do a survey of "test definitions".
>
> I said at the meeting that this would not be an onerous field-by-field survey, but
> rather something simple.
>

Hi Tim,

Here are answers for test-definitions:

> Can everyone who has a test framework or test system please send me a link
> to a "characteristic" test in your system?  Preferably, send me 3 links:
>  - to a "simple" test - one that has few fields but is easy to understand

https://github.com/Linaro/test-definitions/tree/master/automated/linux/busybox

>  - to a "characteristic" test - one that uses the most common features of your system

https://github.com/Linaro/test-definitions/tree/master/automated/linux/libhugetlbfs

>  - to your repository of tests - hopefully a single directory at a repository online where
> I can find multiple test definitions.

We recently moved our 'master' repository to github:
https://github.com/Linaro/test-definitions/

For the 'files' part, each test in test-definitions is stored in a
separate directory. The directory has to contain at least the YAML
file that is compliant with LAVA test definition. We have a sanity
check script (validate.py) that is executed on any pull request. This
ensures all files pushed to the repository are compliant. Usual
practice is that test directory contains a test script (shell script).
Script is responsible for installing dependencies, running tests and
parsing results. There is no mandatory format there but
test-definitions provides a library of functions that help writing
test scripts. There are libraries for 'linux' and 'android'. We also
host directory for manual tests and simple executor for them but in
the context of automated testing these are irrelevant.

Best Regards,
milosz

>
> I'll try to cobble together the list of file formats, fields, operations, and possible values
> based on looking at your code, and the previous survey responses.
>
> I've put some representative data (at least the links and files) for Fuego on the
> page where I'm collecting this information, which is at:
> https://elinux.org/Test_definition_survey
>
> Thanks and regards,
>  -- Tim
>
> --
> _______________________________________________
> automated-testing mailing list
> automated-testing at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/automated-testing