[yocto] [RFC] Yocto Autobuilder and LAVA Integration

[richard.purdie at linuxfoundation.org]

Hi Anibal,

On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
> We know the need to execute OE testimage over real HW not only QEMU, 
> 
> I'm aware that currently there is an implementation on the Yocto
> Autobuilder Helper 
> , this initial implementation looks pretty well separating parts for
> template generation [1] and the script to send jobs to LAVA [2].
> 
> There are some limitations.
> 
> - Requires that the boards are accessible trough SSH (same network?)
> by the Autobuilder, so no distributed LAB testing.
> - LAVA doesn't know about test results because the execution is
> injected via SSH.
> 
> In order to do a distributed Boards testing the Yocto Autobuilder
> needs to publish in some place accessible the artifacts (image,
> kernel, etc) to flash/boot the board and the test suite expected to
> execute. 
> 
> Currently there is a functionality called testexport (not too
> used/maintained) that allows you to export the test suite.

I continue to have mixed feelings about testexport. It adds complexity
but I'm not sure its actually worth it.

An alternative would be to specify a set of commit hashes for the
configuration under test (poky or oe-core+bitbake and any other
layers), then have LAVA obtain those pieces and run the tests directly.

Its worth considering that we already now have two difference pieces of
code trying to package up the build system/layers, eSDK and testexport.
Ideally if we had some kind of standardised layer setup/configuration
approach we'd then just have a config file to share, then the tools
could recreate the environment and allow the tests to be run there
without testexport. Layer-setup is itself a harder subject but for
example the layer setup code in autobuilder-helper could easily be
reused as things stand today...

In fact the more I think about it, the more I think we may want to do
it that way...

> I created a simple LAVA test definition that allows run testimage
> (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> get target and server IP addresses.
> 
> In this way the LAVA server handles all the testing and finally the
> Yocto Autobuilder can get/poll an event to know what was the actual
> result of the job and the job could be send to different LAVA LAB's. 

That does sound useful and is likely a way we could end up doing this.
Its probably worth highlighting that we now have a way of summarising
the result of the test in the form of the json file the tests all
generate. Sharing that back to the Yocto autobuilder would give us the
test results we need.

> Some of the tasks, I identified,  (if is accepted)
> 
> - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior
> , move the EXTRA_PLAIN_CMDS to a class.
> -  Yocto-aubuilder-helper: Create a better approach to re-use LAVA
> job templates across boards.
> - Poky/OE: Review/fix test-export or provide other mechanism to
> export the test suite.

I think some of these are also independent of each other and good
things to work on regardless...

I would like to hear feedback from those at Intel using LAVA who
submitted the existing code.

Cheers,

Richard