[Automated-testing] Automated testing update

[Paul Eggleton]

Hi folks,

Just to give everyone an update of where we are in the Yocto Project with 
automated QA:

1) For running tests on real hardware, we're continuing along the path for the 
1.6 release to get some basic testing capabilities. Stefan has sent patches to 
enable building simple master images, deploy test images using it, and boot 
into the master image (at least for x86 systems; I'm still waiting on the BSP 
for it, but I hope to enable the MIPS EdgeRouter Lite for 1.6 as well). 
Cristian is working on an expect script for the power strip in the test rack 
that we have to be used in conjunction with conmux/hardreset for resetting 
boards.
(To clear up any confusion, we're not expecting to develop this much further 
beyond 1.6 - as I've talked about before, we needed something to allow our QA 
team to streamline their processes without being too disruptive, with the aim 
of looking at a migration to LAVA for the test automation in 1.7. We do expect 
that lessons learned and even some of the scripts will be re-usable in 
conjunction with LAVA though.)

2) The ability to run our tests on a nominated machine if it has the target 
image already installed and you know its IP address has been merged. Whilst by 
no means a complete solution for automated testing, for platforms where we 
can't yet do the image deployment automatically it still enables QA to run the 
rest of the tests more quickly after doing the deployment manually.

3) The ability to export tests so they can be run outside of the build system 
is now merged. The expectation is this will make it easier to run our existing 
tests under the control of an external system (e.g. LAVA) - the build system 
finishes building the images, exports the test scripts and then the scripts can 
be sent off to another system to be run asynchronously. (The exported scripts 
don't handle any deployment, on the understanding that this would be handled 
by other means e.g. by LAVA.) This will also help where your test 
infrastructure is not directly connected to the build machine, which is 
certainly true of our setup.

4) Not directly related, but oe-selftest has also been merged, so we now have 
a framework for testing the build system itself, as well as a few initial 
tests.

As always, the code for this can be found in OE-Core / Poky master under 
meta/lib/oeqa/.

Re LAVA, I've been very busy with 1.6 work lately so I haven't had much of a 
chance to look at it since our last discussion, but I hope to have time soon. 
Does anyone else have anything to report?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre