[poky] distcc/ccache?

[Darren Hart]

On 12/08/2010 03:12 AM, Richard Purdie wrote:
> On Wed, 2010-12-08 at 01:51 -0500, Jon Masters wrote:
>> Anyone happen to know if poky plays nicely with distcc?
>
> I think most success has been with icecc in the past and there is a
> icecc class which has code to enable that. It turns out that compiling
> isn't the main bottleneck in builds though and icecc isn't something we
> test in Poky at this time, we just provide the same class OE has which
> may or may not work.

While not distributed compiling related, the following might add some 
perspective as to what to expect with more compile nodes.

I tried a poky-image-sdk (qemux86_64) build on a 4 socket Westmere 
machine (40 physical cores total, 80 hardware threads). I set my 
local.conf to:

BB_NUMBER_THREADS = "160"
PARALLEL_MAKE = "-j 160"

After a warm-up build (get everything downloaded) I cleaned tmp and 
sstate-cache and did another build. This one build took 118 minutes to 
complete.

$ time bitbake poky-image-sdk

<time passes & bitbake spews messages like g++ and a buggy stl program>

NOTE: Tasks Summary: Attempted 5619 tasks of which 300 didn't need to be 
rerun and 0 failed.

real    117m56.025s
user    961m28.822s
sys     464m27.318s


I checked the CPU load periodically, often finding it under 10%. While 
not a scientific test, what this tells me is that the build is 
relatively serialized and/or much of the process is blocked on I/O. It 
also suggests that a really big machine isn't all that helpful in 
speeding up builds (although 117m is nothing to be ashamed of!).

One could likely run four parallel builds on this system and have them 
complete in about the same amount of time. I ran 4 copies of the above 
load (even leaving the parallelism settings at 160). All 4 builds 
completed in between 186 and 192 minutes. The system load average about 
120 (just eyeballing it), with peaks as high as 700. All four builds 
used the same disk, suggesting build dependencies leading to build 
serialization is the primary bottleneck. So while you definitely want a 
good number of CPUs to build, that number tapers off rapidly. Somewhere 
around 10 I'd guess (for single builds). I'd have to run a number of 
more tests to be sure.

Anyway, hopefully that adds a datapoint of some use.

-- 
Darren Hart
Yocto Linux Kernel