[Automated-testing] LTP and test metadata

[Tim.Bird at sony.com]

> -----Original Message-----
> From: daniel.sangorrin at toshiba.co.jp
> 
> > -----Original Message-----
> > From: Cyril Hrubis
> > Sent: Thursday, August 15, 2019 1:25 AM
> >
> > Hi!
> > As promised I've continued to work on the test metadata
> > export/extraction and apart from hacking on actuall code I've done the
> > most important part, wrote a README where I put all my ideas I've come
> > to when I was keeping the implementation on the backburner.
> >
> > You can see both the new proof of concent along with the documentation
> > at:
> >
> > https://github.com/metan-ucw/ltp/tree/master/docparse
> >
> > Ideally I would love to get some feedback before I return to hacking on
> > the code.
> 
> Thanks for your work.
> 
> Please check my summary and comments.
> 
> 1) Test dependency metadata is stored on each test case (.c file) inside a
> "tst_test" structure. Dependencies are specified through variables that start
> with "needs_" such as needs_root, needs_kconfigs, or needs_tmpdir.
> https://github.com/linux-test-project/ltp/blob/master/include/tst_test.h
> https://elinux.org/Test_Dependencies
> https://github.com/linux-test-
> project/ltp/blob/master/testcases/kernel/syscalls/mkdir/mkdir05.c
> 
> [Comment] perhaps it would be cleaner to separate metadata from actual
> data and functions. For example, by adding a new element to "tst_test"
> called "metadata" that points to an array of "struct tst_metadata". These
> tst_metadata structs would contain the "needs_xxx" configuration options.
> - Using an array would allow specifying slightly different requirements
> depending on the test case variant or parameters.
> - Separating the metadata would also simplify parsing because you would not
> need to filter out non-metadata elements.
> - Downside: the change might be a bit intrusive.
> 
> [Comment] Probably we need to specify some requirements in a parametric
> way. For example: "this test needs 10MB*num_threads memory", where
> num_threads is a test case parameter or another element.
That's interesting, and probably useful for some cases.  But I wouldn't worry
about that in version 1 of this feature.  I think it might unnecessarily complicate
getting the feature off the ground.

I would consider the ramifications of handling paramterized dependencies,
and try not to do anything to preclude them in the future, but I wouldn't
try to support them at the moment.

> 
> [Comment] Sometimes, you may need to specify requirements conditionally.
> For example, if you pass parameter "-s", then you need "root" permissions.
> The same can happen with test variants.

Same comment here.  Adding condition dependencies is probably too difficult
for the first iteration of this.  Having said that, I will admit that this flexibility is
supported in Fuego by allowing imperative code in the test_pre_check function.
But that can't be handled in an automated way, which is what we're striving for
here.

> 
> [Note] here I am assuming that you can pass parameters to the test cases
> (e.g.: size, number of times, units, number of threads etc..) and that the set
> of parameters could affect the dependencies/requirements.
> 
> 2) Test writers can add additional metadata by writting comments in a specific
> format layout. The format layout is an open question.
> 
> [Comment] Could you describe the format that you support in your proof-of-
> concept?
Agreed.  It would be nice to have a description of the format, or some examples.

 -- Tim