[yocto] Discussion: Package testing

[Richard Purdie]

On Thu, 2012-06-14 at 15:26 +0200, Björn Stenberg wrote:
> Many source packages include their own package test suites. We are
> looking at running these tests on target in a structured way, and I
> would like a discussion about how to best fit this into the Yocto
> framework.
>
> The actions that need to be performed for a package test are roughly:
>
> 1) Build the test suite
> 2) Make the test suite appear on target
> 3) Run the test suite
> 4) Parse the results
>
> Each action can be done in several ways, and there are different
> considerations for each solution.
>
> 1) Build the test suite
> -----------------------
> Many package tests are simply bash scripts that run the packaged
> binaries in various ways, but often the test suites also include
> binary tools that are not part of the normal package build. These
> tools have to be built for package testing to work.
>
> Additionally, many packages build and run the test in a single
> command, such as "make check", which is obviously unsuitable for
> cross-compiled packages.
>
> We can solve this in different ways:
>
> a) Run the test build+run jobs on target. This avoids the need to
> modify packages, but building code on target can get quite expensive
> in terms of disk space. This in turn means many tests would require a
> harddisk or network disk to run.
>
> b) Patch the makefiles to split test building and test running.
> Patching makefiles mean we get an additional maintenance and/or
> upstreaming burden, but we should be able to do this in ways that are
> acceptable to upstream. This is our suggestion.

There are some good questions here :)

I think in general, we should be aiming for b) since that falls well
into some of the ideas below. Upstreams should be amenable to splitting
these into two targets (assuming "make check" just runs one after the
other) so we shouldn't have to carry patches in most cases long term.


> 2) Make the test suite appear on target
> ---------------------------------------
> The test suite and utilities obviously have to be executable by the
> target system in order to run. There are a few options for this:
>
> a) Copy all test files to the target at test run time, from the build
> dir, using whatever means available (scp, ftp etc). This limits
> testing to targets/images with easy automatic file transfer abilities
> installed.
>
> b) NFS mount the build dir to access the full work dir and hence the
> test code. this limits testing to targets (and images) with network
> +nfs support. Plus it blends the build env and runtime env in an
> unpleasant way.
>
> c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that
> contain all test files and add those to the image and hence the
> rootfs. This is our suggestion.

I like the idea of test packages.

> 
> 3) Run the test suite
> ---------------------
> Depending on how the test files are presented to the target, the way
> we run them can take different shapes:
>
> a) (scp) A top-level run-all-tests.sh runs on the build host, copies
> all test files from build dir to target, logs in and runs each test.
>
> b) (nfs) run-all-tests.sh is executed in the nfs-mounted build dir on
> target and build-runs each test in its work dir.
>
> c) (-ptest) Install all test files to /opt/ptest/${PN}-${PV} (for
> example). Make a package "ptest-runner" that has a
> script /opt/ptest/run-all-tests to iterate over all installed tests
> and run them. This is our suggestion.

Can we suitably extract the tests out of the source code to be able to
do this? I'm worried a little about the maintenance burden but I do like
option c). 

> 4) Parse the test results
> -------------------------
> Just running the tests doesn't give us much. We have to be able to
> look at the results and make them meaningful to us on a system-global
> level. Packages present their test results in very different ways, and
> we need to convert that to a generic format:
>
> a) Patch each package test to produce a generic ptest output format.
> This is likely difficult to get accepted upstream.
>
> b) Patch the test code minimally and instead use a simple per-packet
> translate script that converts test suite output from the
> package-specific format to a generic ptest format. This is our
> suggestion.
>
> Opinions?
> 
I think the approach described is along the right lines. We'll probably
have to try it and see how it goes. Over time, if upstreams see value in
standard formats and so on, we may be able to effect some change but
right now, we need to start somewhere!

I firmly believe working on these areas will add a lot of value to the
project and attract many more users, as well as help Linux in general so
its something we should work on.

Cheers,

Richard