[Automated-testing] LTP and test metadata

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Cyril Hrubis on Friday, August 23, 2019 3:36 AM
> 
> Hi!
> > Sure, I can add it to the README as well, at this point it's just 1:1
> > serialization of the C structure.
> 
> And I've just pushed an update to the README that contains examples of
> the C code and resulting JSON.

Sorry for these threads that have weeks or months between messages.

But here's some more feedback, now that the json is posted.
These are just some random ideas and feedback on the proposal.

First:
The top-level key for the testcase meta-data is the C filename:

"testcaseXY.c": {
  "needs_root": "1",
 ...
    " Consisting of several lines."
  ],
  "fname": "testcases/kernel/syscalls/foo/testcaseXY.c"
}

I think it would be better to a name not derived from the source.
Instead of <something>.c, can it just be <something>.  (eg. 'abort01')

I think the meta-data name should be the  same as the testcase name that
is used in the test output, so that the description can be matched to a
test result when a particular test fails.

I see you've got an 'fname' field inside the meta-data json, so that can
be used to find the source file that this came from, if needed.

Second:
Test variants are now stored in the runtest files, and one of your open points
for the docparse system is whether to store these in the test somewhere, and
reflect them in the meta-data.

I'm not positive, but my inclination is to say 'yes'.  Fuego stores variants in a separate
file called  the 'spec' file, and documents test variables (knobs that a user can adjust
for different test results) in a test.yaml file.  There are pros and cons to having this
data in a separate file, but in general, I think there are not so many variants that 
it's worth keeping the variants in their own file.  What we tend to do in Fuego
is make it so that the variants are parameterized, and put the values for
such parameters in a board-specific file.

Also, you could actually start with "embedded" variants (that is, keep them inline
in the test), and add support for additional variants from separate files, if that
flexibility is needed in the future.

One class of variant that is often desirable is a "short" one - that runs a particular
test in under 2 minutes.  I might be nice to come up with a standard set of variant
names, that testrunners could select from, to use with a test.

Third:
I think we will definitely shake out the issues with the field names and structure
definition for this meta-data as more systems attempt to generate the data and
use the data.  I'll put it on my to-do list to try to generate a similar file from 
Fuego tests, to see the issues I run into.

Regards, 
 -- Tim