[yocto] Build time data

[Darren Hart]

On 04/12/2012 07:08 AM, Björn Stenberg wrote:
> Darren Hart wrote:
>> /dev/md0        /build          ext4 
>> noauto,noatime,nodiratime,commit=6000
> 
> A minor detail: 'nodiratime' is a subset of 'noatime', so there is no
> need to specify both.

Excellent, thanks for the tip.

> 
>> I run on a beast with 12 cores, 48GB of RAM, OS and sources on a
>> G2 Intel SSD, with two Seagate Barracudas in a RAID0 array for my
>> /build partition. I run a headless Ubuntu 11.10 (x86_64)
>> installation running the 3.0.0-16-server kernel. I can build
>> core-image-minimal in < 30 minutes and core-image-sato in < 50
>> minutes from scratch.
> 
> I'm guessing those are rather fast cores? 

They are:
model name	: Intel(R) Xeon(R) CPU           X5680  @ 3.33GHz


> I build on a different type
> of beast: 64 cores at 2.1GHz and 128 GB ram. The OS is on a single
> SSD and the build dir (and sources) is on a RAID0 array of Intel 520
> SSDs. Kernel is the same ubuntu 3.0.0-16-server as yours.

Now that I think about it, my downloads are on the RAID0 array too.

One thing that comes to mind is the parallel settings, BB_NUMBER_THREADS
and PARALLEL_MAKE. I noticed a negative impact if I increased these
beyond 12 and 14 respectively. I tested this with bb-matrix
(scripts/contrib/bb-perf/bb-matrix.sh). The script is a bit fickle, but
can provide useful results and killer 3D surface plots of build time
with BB and PM on the axis. Can't seem to find a plot image at the
moment for some reason...

> 
> Yet for all the combined horsepower, I am unable to match your time
> of 30 minutes for core-image-minimal. I clock in at around 37 minutes
> for a qemux86-64 build with ipk output:
> 
> ------ NOTE: Tasks Summary: Attempted 1363 tasks of which 290 didn't
> need to be rerun and all succeeded.
> 
> real    36m32.118s user    214m39.697s sys     108m49.152s ------
> 
> These numbers also show that my build is running less than 9x
> realtime, indicating that 80% of my cores sit idle most of the time.

Yup, that sounds about right. The build has a linear component to it,
and anything above about 12 just doesn't help. In fact the added
scheduling overhead seems to hurt.

> This confirms what "ps xf" says during the builds: Only rarely is
> bitbake running more than a handful tasks at once, even with
> BB_NUMBER_THREADS at 64. And many of these tasks are in turn running
> sequential loops on a single core.
> 
> I'm hoping to find time soon to look deeper into this issue and
> suggest remedies. It my distinct feeling that we should be able to
> build significantly faster on powerful machines.
> 

Reducing the dependency chains that result in the linear component of
the build (forcing serialized execution) is one place we've focused, and
could probably still use some attention. CC'ing RP as he's done a lot there.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel