[Automated-testing] LTP and test metadata

[Cyril Hrubis]

Hi!
> > > Sure, I can add it to the README as well, at this point it's just 1:1
> > > serialization of the C structure.
> > 
> > And I've just pushed an update to the README that contains examples of
> > the C code and resulting JSON.
> 
> Sorry for these threads that have weeks or months between messages.

Not a big problem, it's just a bit more expensive to pick up the
context...

> But here's some more feedback, now that the json is posted.
> These are just some random ideas and feedback on the proposal.
> 
> First:
> The top-level key for the testcase meta-data is the C filename:
> 
> "testcaseXY.c": {
>   "needs_root": "1",
>  ...
>     " Consisting of several lines."
>   ],
>   "fname": "testcases/kernel/syscalls/foo/testcaseXY.c"
> }
> 
> I think it would be better to a name not derived from the source.
> Instead of <something>.c, can it just be <something>.  (eg. 'abort01')

For LTP that would just mean stripping the filename extension in 99% of
the cases. And the rest of the names are just variants that are
differentiated with a suffix appended to the name. So this would be
fairly easy to fix, I will look into that.

> I think the meta-data name should be the  same as the testcase name that
> is used in the test output, so that the description can be matched to a
> test result when a particular test fails.

I guess that we would need something that is guaranteed to be unique if
we wanted to match the result to metadata. So either we introduce some
kind of hash that is embedded both in the test and the json or we make
sure that the name is unique when we generate the json file.

> I see you've got an 'fname' field inside the meta-data json, so that can
> be used to find the source file that this came from, if needed.
>
> Second:
> Test variants are now stored in the runtest files, and one of your open points
> for the docparse system is whether to store these in the test somewhere, and
> reflect them in the meta-data.
> 
> I'm not positive, but my inclination is to say 'yes'.  Fuego stores variants in a separate
> file called  the 'spec' file, and documents test variables (knobs that a user can adjust
> for different test results) in a test.yaml file.  There are pros and cons to having this
> data in a separate file, but in general, I think there are not so many variants that 
> it's worth keeping the variants in their own file.  What we tend to do in Fuego
> is make it so that the variants are parameterized, and put the values for
> such parameters in a board-specific file.
> 
> Also, you could actually start with "embedded" variants (that is, keep them inline
> in the test), and add support for additional variants from separate files, if that
> flexibility is needed in the future.

This all sounds good to me. I guess that in 90% cases the variants would
be fairly simple and could be embedded in the special top level test
comment, but there would be cases where it would be easier to put them
into a separate file.

Can you point me to a fuego test variant descriptions?

> One class of variant that is often desirable is a "short" one - that runs a particular
> test in under 2 minutes.  I might be nice to come up with a standard set of variant
> names, that testrunners could select from, to use with a test.

Good idea, I guess that it may have some sense to divide tests into
functional and stress tests as well. Then you can, from a top level
view, say "I want to stress test kernel timers" or "Run a functional
tests for SysV IPC".

> Third:
> I think we will definitely shake out the issues with the field names and structure
> definition for this meta-data as more systems attempt to generate the data and
> use the data.  I'll put it on my to-do list to try to generate a similar file from 
> Fuego tests, to see the issues I run into.

Sounds good :-).

I was also thinking that the json file should probably get some top
level information something that describes the testsuite, it's version,
etc. Something as:

{
	"testsuite": "LTP",
	"url": "https://github.com/linux-test-project/ltp",
	"version": "git:61312c62a392",
	...
	"testcases": [
		...
	]
}

-- 
Cyril Hrubis
chrubis at suse.cz