[Automated-testing] conventions for test invocation/execution, etc.

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Cyril Hrubis 
> 
> Hi!
> This is just an idea, but why don't we standartize on a test description
> instead of on a shell script name? If we choose right format, it would
> be backward compatible and we can add more information there later on...

I think that's a great idea!  In fact, I a variation of that idea independently
when I woke up this morning.  (I think my sub-conscious is actually smarter
than I am.  It figures stuff out while I'm sleeping.  :-)

In any event, I'll write up some of my concepts, which includes having the
test description indicate the test program or test script name, and put them
on the list for discussion (as well as discuss them with people next week
at Plumbers and the CKI hackfest).

Just a quick summary - I think it would be nice to have a common mechanism
to query a test for:
 - dependencies
 - variant name and mechanisms
 - the test program name (or invocation method)
 - result format
 - maybe other info (like that used by LTP to indicate relevance to commit ID, CVEs,
   how to interpret results, etc.)

Despite how heavyweight this sounds, I think it could be done more easily
than one would think.

I think that what LTP is doing with your dependency and test information
tooling could be the seed for this type of feature.  A key notion is that this
information is not carried around independent of the test (like a lot of
frameworks do today), but that the test or its source could be queried
(or scanned) for it.

I'll post more on this later.

Thanks,
 -- Tim