[Automated-testing] Board Farmer [starting]

[Neil Williams]

On Mon, 3 Sep 2018 at 04:22, Chan, Aaron Chun Yew
<aaron.chun.yew.chan at intel.com> wrote:
>
> Hi Ioannis,
>
>
>
> You may want to look into https://www.linaro.org/engineering/projects/lava/ as Yocto Project does uses LAVA for their automated hardware tests/steps.
>
> We at Intel are setting up our Open Source automated hardware tests on reference platform testing related to x86 and x86_64 for Yocto.
>
>
>
> I believe the same model which your trying to achieve can be done in Yocto Project but some work is required at your end to enable building KANO OS
>
> In Yocto. Yocto have layers which supports your model.
>
>
>
> http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/
>
> http://git.yoctoproject.org/cgit/cgit.cgi/meta-qt4/
>
> http://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/
>
>
>
> Regards,
>
> Aaron
>
>
>
> Hey farmers,
>
>
>
> I am a new member on this list. I was introduced to MuxPI and was happy to
>
> meet Paweł Wieczorek in FOSDEM and discuss about it.
>
>
>
> At the moment we haven't set up anything and I am looking around to new
>
> ideas on how to create an OS testing farm for our KANO OS (based on
>
> Raspberry Pi and Debian/Raspbian).
>
>
>
> Please feel free to include my name on the eLinux board farm document,
>
> under the interested people (that would be ioannis at kano.me).
>
>
>
> What would you suggest as a first option to follow CI from Jenkins (where
>
> the image is built) until going further and creating a test farm that can
>
> go through the OS steps (some of them based on a graphical Qt-based
>
> introductory section)? I am open to suggestions and ideas.

Yes, as Aaron mentions, LAVA can provide automated testing of OS
elements on real (and emulated) hardware, concentrating on automated
testing of the kernel. For most devices, especially U-Boot, devices,
this does not generally include testing new builds of U-Boot itself
(due to hardware constraints) but the other components in the stack
can all be tested. When considering automated CI like this, remember
to only test one element at a time. You'll need test jobs to test that
the kernel boots (using an unchanging ramdisk), such as KernelCI does
(https://kernelci.org/). You'll need test jobs which use a kernel
which is known to boot with a rootfs which is new. Then you will get
to the point of testing elements above the bare POSIX shell. There are
a range of tools which can take control of a graphical interface but
success with these tools under full automation is variable. You
certainly don't want to be relying on only being able to test the GUI
to be able to say that the build is good or bad. Testing a GUI adds
complexity and complexity always lowers reliability.

>
>
>
> Things like https://github.com/resin-io/autohat are very interested for my
>
> case! Thank you everybody.

>From a quick glance, that looks like it is limited to running the
tests only in emulation, not on real hardware. If you are making
changes to the kernel then you should test at least the kernel on real
hardware. If you are looking for performance benchmarks or timing
statistics or power consumption optimisation then you will also need
to test on real hardware.

There is no reason to force your testing into a single framework. If
one framework enables GUI testing more easily than another which
provides better kernel testing, then use both. Automation exists to
allow you to make the test matrix as big as you need and two separate
frameworks could work side by side, as long as you provide enough
hardware to run it all (and staff to maintain that hardware).
Automation needs to be reliable for this to work - you have to know
that you can submit 20 or 30 test jobs per build and get reliable
data, within sensible time limits, without humans having to actively
monitor or hand-hold the framework. Having said that, setting up such
a CI framework is a lot of work and there are significant barriers,
non-trivial financial costs and steep learning curves. Start small and
grow slowly. Don't expect to get a usable system for a few months,
minimum. In particular, adding your particular hardware to a CI
framework can be a very difficult task. Running the framework also
needs proper staffing - you will need to allocate people as long term
admins to maintain the infrastructure and manage the test devices.

There is no single automation or CI framework to meet all needs and it
is a mistake for any team to think that their software can meet all
use cases.

The further you get up the software stack (counting the bootloader &
kernel as the bottom), the less useful it becomes to test with real
hardware versus emulation. If you are only interested in testing the
GUI then a fully emulated framework will be easier to maintain.

-- 

Neil Williams
=============
neil.williams at linaro.org
http://www.linux.codehelp.co.uk/