[Automated-testing] Test definition survey

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Milosz Wasilewski 
> 
> On Fri, 11 Jan 2019 at 01:41, <Tim.Bird at sony.com> wrote:
> >
> > OK everyone.  My holidays are over, and it's time to roll up my sleeves and
> get back to work
> > on automated testing standards.
> >
> > One of my action items from the Automated Testing Summit
> > (see https://elinux.org/Automated_Testing_Summit)
> > was to do a survey of "test definitions".
> >
> > I said at the meeting that this would not be an onerous field-by-field
> survey, but
> > rather something simple.
> >
> 
> Hi Tim,
> 
> Here are answers for test-definitions:
> 
> > Can everyone who has a test framework or test system please send me a
> link
> > to a "characteristic" test in your system?  Preferably, send me 3 links:
> >  - to a "simple" test - one that has few fields but is easy to understand
> 
> https://github.com/Linaro/test-
> definitions/tree/master/automated/linux/busybox

It will be useful to compare your busybox test to Fuego's (and maybe
to Yocto Project's, to see how they differ.

> 
> >  - to a "characteristic" test - one that uses the most common features of
> your system
> 
> https://github.com/Linaro/test-
> definitions/tree/master/automated/linux/libhugetlbfs
> 
> >  - to your repository of tests - hopefully a single directory at a repository
> online where
> > I can find multiple test definitions.
> 
> We recently moved our 'master' repository to github:
> https://github.com/Linaro/test-definitions/
> 
> For the 'files' part, each test in test-definitions is stored in a
> separate directory. The directory has to contain at least the YAML
> file that is compliant with LAVA test definition. We have a sanity
> check script (validate.py) that is executed on any pull request.

That's a nice bit of test system validation!  I've been meaning
to do something like that for Fuego.  I hope you don't mind if 
I 'crib' from your example. :-)

> This
> ensures all files pushed to the repository are compliant. Usual
> practice is that test directory contains a test script (shell script).
> Script is responsible for installing dependencies, running tests and
> parsing results. There is no mandatory format there but
> test-definitions provides a library of functions that help writing
> test scripts. There are libraries for 'linux' and 'android'. We also
> host directory for manual tests and simple executor for them but in
> the context of automated testing these are irrelevant.

Thanks very much!  I'll take a closer look at the details next week.
 -- Tim