[Automated-testing] [EXTERNAL] RE: ELC followup / Search for a testing framework for small distribution tests on device

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Westermann, Oliver 
> 
> > On Wed, 13 Nov 2019 at 06:02, Tim.Bird at sony.com <Tim.Bird at
> sony.com> wrote:
> > Fuego is designed to be a host/target embedded test system.  It comes
> with
> > a suite of existing tests.  It also comes with a Jenkins system for controlling
> > everything.  However, tests can also be invoked from the command line
> > independent of Jenkins.  This was done specifically to allow integration with
> > other CI frameworks.
> 
> Assuming one would like to play around with fuego, is there something like a
> referece setup that is easy to reproduce? So eg "Setup a Pi/BB, setup this
> container(s), run "start_test.sh", so we could take a look at Fuego without
> digging through documentation?

While we don't have an official reference setup, many of our examples
use a BeagleBone black board, and there is a sample configuration for
such a board included in the repository.  See the file fuego-ro/boards/bbb.board
for examples.

There are two quick-setup documents available:
 * http://fuegotest.org/wiki/Quick_Setup_Guide
 * http://fuegotest.org/wiki/Fuego_Quickstart_Guide

The first one is smaller, and sets up a "fake" board using the existing
docker container.  It is intended to let people use Fuego for a quick
trial to see what it looks like.

> 
> > Fuego does not currently do board provisioning or boot-time tests,
> > although some users of Fuego have their own Jenkins jobs to perform
> > those steps.
> 
> That might be fine for us.
> 
> > Fuego runs tests from the host (putting all test materials on the target prior
> > to each test, and removing them at the completion of the test).
> 
> How to you make sure that there is no cross-test effects? Leftover temp
> files, bad
> system state, ...?

We have several lines of defense here:
 - Each test is supposed to clean up after itself, by restoring manipulated
  items and removing things that were added
 - Fuego itself removes the entire test directory, and kills processes related to the test
 - Fuego allows a test to specify that the board should be rebooted after the test
  (but the default is not to reboot)
 - If the board is hung, Fuego will detect this and reboot the board
  (this requires that the board power control be configured for the board)

There is always the possibility that a test will corrupt the environment in a persistent
way.  Detecting this and re-provisioning the board is left to the lab administrator
or the provisioning layer of the CI system.

> 
> > IMHO, Fuego's strengths are in generic distro testing and it's existing test
> suite.
> 
> Which might be a very good starting point as to "what to test beside the
> obvious".
> Is there a overview of the tests (or a link to a repo thats "human readable")?
> I struggled
> when googling for what Fuego tests exactly.

Once installed, you can do 'ftc list-tests', or look at:
the fuego-core/tests directory.  Many (but not all) tests
have a test.yaml file which describes the test and its attributes.

You can browse the tests online here:
https://bitbucket.org/fuegotest/fuego-core/src/master/tests/

If you install it and try it out, let me know your impressions (good or bad).
Especially let me know if something didn't work as expected.
It's always good to get feedback.

Regards,
 -- Tim