[yocto] RFC: automated runtime testing on real hardware

[Darren Hart]

On Tue, 2013-08-13 at 12:00 +0100, Paul Eggleton wrote:
> Hi all,
> 
> As part of the feature development work for 1.5, we've recently added the 
> ability to easily define runtime tests written in python, replacing the old 
> shell-based imagetest.bbclass. However, for 1.5 this is limited to running the 
> tests within a QEMU environment; the obvious and desirable next step is to be 
> able to have these run on real hardware. Clearly this presents some 
> challenges, such as how to manage access to the hardware if you have multiple 
> builders potentially wanting to use the same board at around the same time.
> 
> Automated testing on real hardware is something that a lot of us do already, 
> but the likelihood is that we've all been coming up with our own solutions for 
> this, so it would be really great if we could have a single working solution 
> to cover the generally applicable use cases. As a start though it would be 
> interesting to hear from people who have existing test setups or who are 
> generally interested in this area. I'd like to hear feedback on the following:
> 
>  * What hardware setup do you use for automated testing?

I am starting to put something together using a customized 19" rack
along the lines of those used by OSADL for their real-time farm:

https://www.osadl.org/Test-Rack.test-rack.0.html

This is a poor man's BladeCenter for embedded boards in that it provides
a common interface to each board for the rack infrastructure. The board
specific stuff sits on the trays themselves. The trays are easily
removable to take a board to a desk for hands-on work.

A smart-switch, web-power strip, and serial concentrator feed into a
"gateway server" where services like conmux glue it all together.

> 
>  * Do you use any software to manage the hardware? Does this include any 
> existing open source tools?

conmux
dnsmasq
udev (for static usb uart naming)

The testrack has some early-stage tools to support it, but I'm still
sorting those out and they aren't particularly robust for initial setup
quite yet.

> 
>  * What would you like to see in a common framework to enable this kind of 
> testing?

To simplify the board management you allude to above, I believe a pull
model, rather than a push, is a better architecture.

Your builders can complete builds and inject test records into a
database with a priority. The boards sit in a
boot/check-db/deploy-test-img/reboot-to-test-image/run-tests loop and
push the results back to the database. They select tests based on
priority, and the builder doesn't have to do anything at all to manage
the boards. This also decouples the builder from the test
infrastructure.

> 
> For reference, some discussion on this started in another thread [1], but I'd 
> like to throw this out to a more general audience.
> 
> Thanks,
> Paul
> 
> [1] https://lists.yoctoproject.org/pipermail/yocto/2013-August/017687.html
> 

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel