[Automated-testing] Test stack survey - LKFT

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Michal Simek 
> On 9.10.2018 01:05, Tim.Bird at sony.com wrote:
> >
> >
> >> -----Original Message-----
> >> From: Mark Brown
> >>
> >> On Mon, Oct 08, 2018 at 09:19:43PM +0000, Tim.Bird at sony.com wrote:
> >>
> >>> OK - this is interesting.  Fuego tries to address this issue with something
> >>> we call the 'fail_ok_list', that lists expected failures.  However we leave
> >>> them as failures, but prevent them from bubbling up the
> >> testcase/test_set/test_suite
> >>> hierarchy to indicate overall test failure.
> >>
> >> FWIW the XFAIL thing is something GCC do as well (or used to last time I
> >> paid much attention to GCC development which was "some" years ago).
> >
> > I'll have to check it out.  Thanks.
> >
> > gcc overall is interesting when viewed from the perspective of it being a
> test
> > (which it is often used for - lots of 'build' tests consist of just executing the
> > compiler).  It's often viewed as a compilation tool, with testing as a side-
> effect.
> > Which is correct.  But if you look at how one uses it to test things, there are
> a
> > number of features:
> > 1) the reporting space is sparse - only failures and warnings are printed -
> not
> > every testcase that is executed.  This is because there can be thousands of
> > 'tests' done on every line of code that is compiled, and the testcase space is
> > just too big.
> > 2) users can control what things are considered errors (so, similar to a
> > fail_ok_list, or the xfail thing we were just discussing), as well as
> > what things are considered warnings (using -W flags).  Sometimes, in-
> source
> > mechanisms are used to mark things to be ignored.
> > 3) They have a fairly well-regularized, line-based output format.  For each
> testcase
> > reported it consists of an error or warning line, with consistent strings, and
> additional
> > diagnostic data always on  subsequent lines (if I'm not mistaken).  They've
> thought
> > about machine parsing of their output.
> 
> Also pytest have xfail listed.
> https://docs.pytest.org/en/latest/skipping.html
> 
> In U-Boot we use test/py/ pytest testing framework and I have marked
> some internal tests as xfail because when we are running the same tests
> on HW and QEMU where not everything is modeled.

Interesting.  They also have something called 'xpass', for when a test that
was expected to fail passes.

I only looked very briefly, but it looks like the xfail status for a test is encoded
in the test itself.  What we have found in Fuego is that you need to be
able to keep the pass criteria separate from the test in some circumstances.
It often needs to be customized per-board, which you can't do if it's embedded in
the test.

This is an interesting approach that I'll have to think about more.
Thanks!
 -- Tim