[intel-iot-refkit] speeding up CI builds

[Patrick Ohly]

On Fri, 2017-05-26 at 06:32 +0000, Kartau, Olev wrote:
> On Wed, 2017-05-24 at 18:09 +0200, Patrick Ohly wrote:
> > Hello!
> > 
> > Olev recently noticed that compressing images can be quite slow (several
> > minutes) and happens to be on the critical path. I looked into disabling
> > the compressed image variants (like wic.xz) for the oe-selftest. But at
> > least for the installer image tests that forces a rebuild of the
> > installer image, which defeats the original goal of using the image that
> > was already built earlier during the main build.
> > 
> > We don't publish images for every pull request test build, do we?
> Builders store all images in internal server (iot-ref-kit.ostc.intel.com,
> same one that hosts buildhistory and caches), where HW-testers
> can fetch images for testing.

Good point, I hadn't thought of that use. We need to test, though,
whether using uncompressed images really brings an overall speedup.

> >  So I
> > think it would be best if we disabled the
> > 
> >         REFKIT_VM_IMAGE_TYPES = "wic.xz wic.zip wic.bmap
> >         wic.xz.sha256sum"
> > 
> in current HW-tester script, that would cause failure as it tries
> to download either wic.xz or wic.zip,
> but that's easy to fix by adding 3rd try for wic.

And perhaps eliminate wic.zip entirely?

> > Build time with PR #147 merged will be around 2:15. This may go down
> > again when updating OE-core (which has a patch that avoids unnecessary
> > image rebuilding), but eventually we are going to have to deal with
> > multi-hour build tests again.
> > 
> > I see some potential solutions:
> >      1. parallelize some of our oe-selftests, like OE-core is currently
> >         doing
> Making post-build tests parallel would be cool, if that's safely doable.
> Now these mostly utilize one core on a builder.
> But is it so these tests need clean state of workspace,
> and parallel run risks to affect same workspace from other tests?

It is definitely not easy, because the tests run much more efficiently
when sharing the same build directory. When parallelizing at a high
level, the individual tests may each end up building the same thing
multiple times in their private build directory. When parallelizing deep
inside a test, the tests become more complicated.

Note that the upstream effort in OE-core is currently limited to tests
that do not involve a bitbake invocation in a build directory,
presumably for similar reasons.

> >      2. run only fast and/or selected (*) selftests for PRs, merge them
> >         into a master-next branch, and only do full testing of
> >         master-next before promoting that to master
> > 
> > I'm not sure how much parallelization will really help us.
> > 
> > (*) with "selected" I meant "selected by the author of the PR". If I add
> > a new feature and have new tests for that, I need a way to trigger those
> > tests for that PR. The mechanism for that would have to be defined and
> > implemented. Perhaps some annotations in the PR description which gets
> > exported to bitbake and thus refkit-ci.inc?
> > 
> CI build sees PR description text in it's env, so this seems to be doable,
> but adds need to invent some format and parsing of it.
> Another channel is Github PR labels, also passed into build.
> Managing labels set for that will be needed then as extra effort.

Labels can usually only be set after creating the PR, and then a test
build is likely already running.

I think including something in the PR description is more promising.

-- 
Best Regards

Patrick Ohly
Senior Software Engineer

Intel GmbH
Open Source Technology Center               
Usenerstr. 5a                           Phone: +49-228-2493652
53129 Bonn
Germany