[Automated-testing] LTP and test metadata

[Dmitry Vyukov]

On Thu, Oct 3, 2019 at 11:35 PM <Tim.Bird at sony.com> wrote:
> > -----Original Message-----
> > From: Cyril Hrubis
> >
> > Hi!
> > > > [Note] here I am assuming that you can pass parameters to the test cases
> > > > (e.g.: size, number of times, units, number of threads etc..) and that the
> > set
> > > > of parameters could affect the dependencies/requirements.
> > > >
> > > > 2) Test writers can add additional metadata by writting comments in a
> > specific
> > > > format layout. The format layout is an open question.
> > > >
> > > > [Comment] Could you describe the format that you support in your
> > proof-of-
> > > > concept?
> > > Agreed.  It would be nice to have a description of the format, or some
> > examples.
> >
> > As I tried to explain, there is no format yet, it's just a text, it
> > depends on how we make use of the data.
> >
> > So we can start by agreeing on the markup format for the test
> > documentation, I would go for one of the asscii based formats but there
> > are a few to choose, the most used ones are probably:
> >
> > * markdown
> > * asciidoc
> > * reStructuredText
> >
> > Preferencies aside markdown seems to be aspiring for a industry
> > standard, mostly due to GitHub so sticking to it would be a logical
> > choice.
>
> Just a note.  At the CKI hackfest we discussed adding meta-data
> to kselftests, and came to the conclusion that for any meta-data
> expressed in the Linux kernel tree inside source files, we're likely
> going to have to use the kernel-doc format (or have to explain
> why we're introducing a new structured-comment format).

If it's expressed as comments, I would suggest to ensure that it's
possible to detect any errors in the descriptions (rather than be
silently confusing mis-spelled metadata with non-metadata and
ignoring). If it's not checked it will slowly get into unmaintainable
mess over years ("oh, turns out we did not run this test for past 3
years, because there was a typo").

An example could be:

/* META
...
... actual metadata goes here
...
*/

And then a checker will ensure that (1) each test has that block, (2)
every line of the block parses as metadata (no silent ignoring).
It may also allow for a more handy for humans format (if it's checked
you don't need some overly verbose format just to be able to
distinguish metadata from non-metadata).