[yocto] Build time data

[Darren Hart]

On 04/12/2012 03:43 PM, Chris Tapp wrote:
> On 12 Apr 2012, at 15:34, Darren Hart wrote:
>>
>>
>> On 04/12/2012 07:08 AM, Björn Stenberg wrote:
>>> Darren Hart wrote:
>>>> /dev/md0        /build          ext4 
>>>> noauto,noatime,nodiratime,commit=6000
>>>
>>> A minor detail: 'nodiratime' is a subset of 'noatime', so there is no
>>> need to specify both.
>>
>> Excellent, thanks for the tip.
>>
>>>
>>>> I run on a beast with 12 cores, 48GB of RAM, OS and sources on a
>>>> G2 Intel SSD, with two Seagate Barracudas in a RAID0 array for my
>>>> /build partition. I run a headless Ubuntu 11.10 (x86_64)
>>>> installation running the 3.0.0-16-server kernel. I can build
>>>> core-image-minimal in < 30 minutes and core-image-sato in < 50
>>>> minutes from scratch.
>>>
>>> I'm guessing those are rather fast cores? 
>>
>> They are:
>> model name	: Intel(R) Xeon(R) CPU           X5680  @ 3.33GHz
> 
> Nice, but well out of my budget - I've got to make do with what one of your CPUs costs for the whole system ;-)
> 
>>
>>> I build on a different type
>>> of beast: 64 cores at 2.1GHz and 128 GB ram. The OS is on a single
>>> SSD and the build dir (and sources) is on a RAID0 array of Intel 520
>>> SSDs. Kernel is the same ubuntu 3.0.0-16-server as yours.
>>
>> Now that I think about it, my downloads are on the RAID0 array too.
>>
>> One thing that comes to mind is the parallel settings, BB_NUMBER_THREADS
>> and PARALLEL_MAKE. I noticed a negative impact if I increased these
>> beyond 12 and 14 respectively. I tested this with bb-matrix
>> (scripts/contrib/bb-perf/bb-matrix.sh). The script is a bit fickle, but
>> can provide useful results and killer 3D surface plots of build time
>> with BB and PM on the axis. Can't seem to find a plot image at the
>> moment for some reason...
>>
>>>
>>> Yet for all the combined horsepower, I am unable to match your time
>>> of 30 minutes for core-image-minimal. I clock in at around 37 minutes
>>> for a qemux86-64 build with ipk output:
>>>
>>> ------ NOTE: Tasks Summary: Attempted 1363 tasks of which 290 didn't
>>> need to be rerun and all succeeded.
>>>
>>> real    36m32.118s user    214m39.697s sys     108m49.152s ------
>>>
>>> These numbers also show that my build is running less than 9x
>>> realtime, indicating that 80% of my cores sit idle most of the time.
>>
>> Yup, that sounds about right. The build has a linear component to it,
>> and anything above about 12 just doesn't help. In fact the added
>> scheduling overhead seems to hurt.
>>
>>> This confirms what "ps xf" says during the builds: Only rarely is
>>> bitbake running more than a handful tasks at once, even with
>>> BB_NUMBER_THREADS at 64. And many of these tasks are in turn running
>>> sequential loops on a single core.
>>>
>>> I'm hoping to find time soon to look deeper into this issue and
>>> suggest remedies. It my distinct feeling that we should be able to
>>> build significantly faster on powerful machines.
>>>
>>
>> Reducing the dependency chains that result in the linear component of
>> the build (forcing serialized execution) is one place we've focused, and
>> could probably still use some attention. CC'ing RP as he's done a lot there.
> 
> Current plan for a 'budget' system is:
> 
> DX79TO motherboard, i7 3820, 16GB RAM, a pair of 60GB OCZ Vertex III's in RAID-0 for downloads / build, SATA HD for OS (Ubuntu 11.10 x86_64).
> 
> That'll give me a 2.7x boost just on CPU and the SSDs (and maybe some over-clocking) will give some more.
> 
> Not sure if SSDs in RAID-0 will give any boost, so I'll run some tests.
> 
> Thanks to all for the comments in this thread.

Get back to us with times, and we'll build up a wiki page.

> 
> Chris Tapp
> 
> opensource at keylevel.com
> www.keylevel.com

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel