[yocto] RFC: automated runtime testing on real hardware

[George Harris]

To this point, our testing strategy uses Yocto, but is not contained within
Yocto. As part of a build, we produce two images: (1) device image, and (2)
image containing everything necessary to run tests against a device. The
second part is important to us, as we want a separation between the device
and tests run against the device. Tests against a device are always
performed across the network, between (1) and (2). We want anyone to easily
be able to run or develop tests without needing to setup a workstation or a
workspace. A separate testing image, runnable in QEMU, allows anyone to
pick up a build and start testing with minimal effort. We also build a
toolchain containing QEMU to ensure that everyone uses the same version and
features of QEMU. Finally, we have many tests, some which take longer than
24 hours to run. We want to parallelize these tests and run them across
multiple systems at the same time. Because of this, we take the output from
a build and run tests. Some tests may make sense to run every build, but
other tests are run periodically throughout the day independent of the
build.

Both our build system and test system are based on Jenkins. One job will
build the code, and another will run tests. The test job is configured to
run multiple types of tests in multiple configurations, all concurrently.

As our tests are always run across the network, tests need not know if the
device under test is a simulation or hardware. Jenkins is used to manage
requests for hardware; if a configuration requires the use of a piece of
hardware, that part of the job blocks until the hardware is available. When
the hardware becomes available, Jenkins will allow that part of the job to
start.

As for the arrangement of the hardware, we have a testbed consisting of
several devices. Each device is connected to the network and hooked up to a
power switch that is connected to the network. When a test is run against a
device, that device is reprogrammed, power cycled, then the tests are run.

To answer some of your specific questions:

 * What hardware setup do you use for automated testing?
>
There is a testbed of devices, each connected to the network with a fixed
IP address. Jenkins is used to run tests, and also manages access to
hardware via labels. When a Jenkins test job requires a piece of hardware
and that hardware is available, a configuration file is used to map the
device to use to its IP address. From there, tests are run against the
device.

 * Do you use any software to manage the hardware? Does this include any
> existing open source tools?
>
The only software we used to manage the hardware is Jenkins. We do use
other software to manipulate the hardware. For example, a power switch
accessible over the network, a DHCP server to assign the proper address,
etc.

Likely this will not help you develop a hardware testing solution in Yocto.
Maybe it will give you a data point though.

- George
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20130815/f2fb5a4c/attachment.html>