[yocto] Details on automated Yocto validation?

[Paul Eggleton]

On Tuesday 15 January 2013 16:46:49 Sandino Flores Moreno wrote:
> Where may I find details about Yocto Validation?
> Especially in what regards to automated tests.
> 
> I came up with this topic because we are using it to build a custom linux
> distribution, rpm based, and now it is required to implement a proper
> validation to ensure there are no regressions, that packages are sane, etc.
> 
> As it normally happens, there may be an effort that someone else started
> already.
> 
> So far I have only found:
> https://wiki.yoctoproject.org/wiki/Yocto_1.4_Schedule
> 
> But it's, as the name states, it is very general.
> Something more specific would be useful.

Sorry for the (extremely) late reply. There are a few different aspects to
testing within the Yocto Project:

* Autobuilder - we have an autobuilder infrastructure upon which we run
regular builds for continuous integration purposes. Our autobuilder instance

http://autobuilder.yoctoproject.org/

The software that is used to manage the autobuilder infrastructure, which is
based upon Buildbot, is also made available so that you can set up your own autobuilder
if desired:

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/

* We also have a QA process for every release. This includes a number of manual
tests which are recorded in Testopia within our Bugzilla instance at
http://bugzilla.yoctoproject.org. We plan to automate many of these tests
as part of the 1.5 release.

* Package QA tests: these are run automatically on every package built by the
build system. This is implemented in classes/insane.bbclass and is now
documented in the reference manual:

http://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#ref-classes-insane

* ptest: in 1.4 we added the "ptest" framework which allows you to integrate
tests for individual software packages and run them on the target hardware (or
within emulation). There is a section within the manual explaining how to use
the ptest framework:

http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#testing-packages-with-ptest

* qemuimagetest: there are a limited set of runtime tests that you can
run automatically on every built image (and we enable these on the
autobuilder). These are not very well-documented but will likely be
rewritten for the next release in any case (1.5); however there is some
information on how to enable these in the sample local.conf - search
for "IMAGETEST". The implementation of the tests can be found in
scripts/qemuimage-tests/.

You can see some of the improvement work we have planned for 1.5 in the
bugs linked from this entry in Bugzilla:

  https://bugzilla.yoctoproject.org/show_bug.cgi?id=4137

Hope this helps.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre