[Automated-testing] Test stack survey - LKFT

[Milosz Wasilewski]

On Mon, 8 Oct 2018 at 18:57, <Tim.Bird at sony.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Milosz Wasilewski
>
> OK - thanks for the response.  I have created a wiki page for this here:
> https://elinux.org/LKFT_survey_response
>
> one or two questions or comments below...
>
> >
> > ==== Test scheduling/management ====
> > Does your test system:
> > * check that dependencies are met before a test is run?
> > "yes - as a part of test job setup"
> > * schedule the test for the DUT?
> > "yes"
> > ** select an appropriate individual DUT based on SUT or test attributes?
> > "yes"
> > ** reserve the DUT?
> > "yes"
> > ** release the DUT?
> > "yes"
> > * install the software under test to the DUT?
> > "yes"
> > * install required packages before a test is run?
> > "leave to user - dependency installation can be part of a test execution"
> > * require particular bootloader on the DUT? (e.g. grub, uboot, etc.)
> > "no"
> > * deploy the test program to the DUT?
> > "yes"
> > * prepare the test environment on the DUT?
> > "yes"
> > * start a monitor (another process to collect data) on the DUT?
> > "no - test monitoring happens over serial connection to DUT"
>
> OK  - is there any support for something like performing
> a trace during execution, or would this be left to the user
> to configure as part of test definition?

I can't imagine how that would work to be honest. LKFT uses LAVA for
executing tests. LAVA executes tests by running shell scripts. If
traces can be collected this way than it can be done. But you're
right, it's left to the user to configure.

>
> > * start a monitor on external equipment?
> > "yes"
> > * initiate the test on the DUT?
> > "yes"
> > * clean up the test environment on the DUT?
> > "no"
>
> I presume this is because LKFT follows the LAVA model
> of rebooting between tests, so the slate is always clean?
> If not, let me know.

yes, LKFT uses LAVA for execution. From my experience not many test
suites cleans after themselves. LAVA doesn't do that for sure.
Rebooting doesn't always guarantee clean start of the following tests.

>
> > ==== Run artifact handling ====
> > Does your test system:
> > * store run artifacts
> > "yes"
> > ** in what format?
> > "leave to user - uploaded files are generated during the test, which
> > is controlled by user"
> > * put the run meta-data in a database?
> > "yes"
> > ** if so, which database?
> > "postgresql"
> > * parse the test logs for results?
> > "yes"
> > * convert data from test logs into a unified format?
> > "no"
> > ** if so, what is the format?
> > * evaluate pass criteria for a test (e.g. ignored results, counts or
> > thresholds)?
> > "yes"
> > * do you have a common set of result names: (e.g. pass, fail, skip, etc.)
> > "yes"
> > ** if so, what are they?
> > "pass, fail, skip, xfail (for ignored failed test)"
> xfail is interesting.  I haven't seen that one called out (unless
> I'm not understanding it correctly)
>
> What category would you assign to a testcase that failed to
> run due to:
> 1) user abort of the test run?
> 2) DUT is not available?
> 3) test program faulted?
>
> Fuego assigns these to 'error', which means something went wrong
> besides the testcase itself.   Does LKFT have a similar result category?
> (Well, sometimes we just leave a testcase result as SKIP if we don't
> have a result for it and can't determine what happened.)

It depends on the cause I think. 'xfail' is assigned to the test case
that was executed properly and produced a 'fail' result. Than during
result postprocessing we check whether such failure is marked as
'known issue' (in database). If it is, result is changed to xfail. I
know this isn't correct usage of this state. IIRC xfail should simply
be equal to 'pass' because we deliberately trigger the condition that
is considered 'failed' for some reason. Anyway, LKFT uses xfail to
mark failures that we know about but don't want to deal with right
now. It is meant do make distinction between 'known' failure and
'regression' which is 'unexpected failure'.

To answer the question above - with LAVA it's very hard to catch this
kind of problem. When test fails to run but subsequent tests are
executed, we simply don't know about that. There is no concept of test
plan (as in 'detailed list of test results that should be produced')
in either LAVA or LKFT. It would be pretty hard to track such cases.
1) user abort of test run - no results are produced. This is unusual
situation as tests should be executed without any interaction
2) DUT is not available - no tests will be attempted. LAVA only starts
execution when DUT is available. If it 'disappears' during execution
LAVA usually aborts the whole test job
3) test program faulted - if subsequent tests can be run, we don't
detect it. If fault results in system crash, cpu freeze or similar,
LAVA job is aborted due to timeout.
So our xfail is none of the above.

>
> > ==== User interface ====
> > Does your test system:
> > * have a visualization system?
> > "yes"
> > * show build artifacts to users?
> > "no"
> > * show run artifacts to users?
> > "yes"
> > * do you have a common set of result colors?
> > "yes"
> > ** if so, what are they?
> > "red - fail
> > green - pass
> > yellow - skip
> > blue - xfail
> > grey - total"
> The separate color for aggregate result counts is also interesting.
>
> > ==== Requirements ====
> > Does your test framework:
> > * require minimum software on the DUT?
> > "yes"
> > * require minimum hardware on the DUT (e.g. memory)
> > "yes"
> > ** If so, what? (e.g. POSIX shell or some other interpreter, specific
> > libraries, command line tools, etc.)
> > "software - POSIX shell
> > hardware:
> >  - boot on power
>
> Good catch!  It's surprising that there are dev boards in this day and age
> that don't support this simple attribute.  It might be good to call this out
> in any documents we produce, as a recommended practice.
> I assume by this, that you mean a board that comes all the way up
> to the functioning state of SUT without requiring button presses, or
> any other user action.  Is that right?

yes, correct. As example of board that doesn't do that - BeagleBoard
X15 from TI. It requires button press to start booting. We had to
modify the boards and add relays to work around that.

>
> >  - serial line with unique ID
> I'm not sure what you mean by this.  Does this mean something
> that can have a unique id in /dev/serial/by-id?

yes. Since LAVA talks to devices using serial line and there can be
multiple devices connected to the same host, serial ID number has to
be unique.

>
> >  - 'not-crazy' bootloader"
>
> Can you be more specific?  (either by naming a 'crazy' bootloader, or
> a bootloader feature (or anti-feature) that would make it 'crazy')?  :-)

I'd say UEFI menu is 'crazy enough' but LAVA can (barely) deal with it
:) All ncurses style menus are 'no no'; color codes in serial output
are also impossible to work with.

milosz

>
> Thanks very much!
>  -- Tim
>
>