[yocto] Discussion: Package testing

[Fredrik Hugosson]

On 06/18/2012 01:43 PM, Richard Purdie wrote:
> On Thu, 2012-06-14 at 15:26 +0200, Björn Stenberg wrote:
>> Many source packages include their own package test suites. We are
>> looking at running these tests on target in a structured way, and I
>> would like a discussion about how to best fit this into the Yocto
>> framework.
>>
>> The actions that need to be performed for a package test are roughly:
>>
>> 1) Build the test suite
>> 2) Make the test suite appear on target
>> 3) Run the test suite
>> 4) Parse the results
>>
>> Each action can be done in several ways, and there are different
>> considerations for each solution.
>>
>> 1) Build the test suite
>> -----------------------
>> Many package tests are simply bash scripts that run the packaged
>> binaries in various ways, but often the test suites also include
>> binary tools that are not part of the normal package build. These
>> tools have to be built for package testing to work.
>>
>> Additionally, many packages build and run the test in a single
>> command, such as "make check", which is obviously unsuitable for
>> cross-compiled packages.
>>
>> We can solve this in different ways:
>>
>> a) Run the test build+run jobs on target. This avoids the need to
>> modify packages, but building code on target can get quite expensive
>> in terms of disk space. This in turn means many tests would require a
>> harddisk or network disk to run.
>>
>> b) Patch the makefiles to split test building and test running.
>> Patching makefiles mean we get an additional maintenance and/or
>> upstreaming burden, but we should be able to do this in ways that are
>> acceptable to upstream. This is our suggestion.
>
> There are some good questions here :)

Yes there are!

> I think in general, we should be aiming for b) since that falls well
> into some of the ideas below. Upstreams should be amenable to splitting
> these into two targets (assuming "make check" just runs one after the
> other) so we shouldn't have to carry patches in most cases long term.
>

I think you both might be missing some cases here. The problem is that 
'make check' does not have a standardized meaning.

 From the top of my head, I can think of these different variants:
1) Function test level with testing run on target, package cross-compiled
2) Function test level with testing run on host, package cross-compiled
3) Function test level with testing run on host, package compiled for host
4) Unit test level with testing run on target, package cross-compiled
5) Unit test level with testing run on host, package compiled for host

I think that we need to be able to separate the different uses of 'make 
check' to be able to support testing nicely.

>> 2) Make the test suite appear on target
>> ---------------------------------------
>> The test suite and utilities obviously have to be executable by the
>> target system in order to run. There are a few options for this:
>>
>> a) Copy all test files to the target at test run time, from the build
>> dir, using whatever means available (scp, ftp etc). This limits
>> testing to targets/images with easy automatic file transfer abilities
>> installed.
>>
>> b) NFS mount the build dir to access the full work dir and hence the
>> test code. this limits testing to targets (and images) with network
>> +nfs support. Plus it blends the build env and runtime env in an
>> unpleasant way.
>>
>> c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that
>> contain all test files and add those to the image and hence the
>> rootfs. This is our suggestion.
>
> I like the idea of test packages.

Would be very nice for functional tests, but it will not be so simple 
for unit testing. For unit tests I belive you need some build target 
too, which leads back to the comment above.

It might be a lot of autoconf hacking to get this working, don't know if 
there are any widely accepted standard targets for it.

/Regards
Fredrik Hugosson