[yocto] Build time data

[Darren Hart]

On 04/18/2012 12:41 PM, Chris Tapp wrote:
> On 12 Apr 2012, at 23:56, Darren Hart wrote:
>> Get back to us with times, and we'll build up a wiki page.
> 
> Some initial results / comments:
> 
> I'm running on:
>  - i7 3820 (quad core, hyper-treading, 3.6GHz)
>  - 16GB RAM (1600MHz XMP profile)
>  - Asus P9X79 Pro motherboard
>  - Ubuntu 11.10 x86_64 server installed on a 60GB OCZ Vertex 3 SSD on a 3Gb/s interface
>  - Two 60GB OCZ Vertex 3s as RAID-0 on 6Gb/s interfaces.
> 
> The following results use a DL_DIR on the OS SSD (pre-populated) -
> I'm
not interested in the speed of the internet, especially as I've only got
a relatively slow connection ;-)
> 
> Poky-6.0.1 is also installed on the OS SSD.
> 
> I've done a few builds of core-image-minimal:
> 
> 1) Build dir on the OS SSD
> 2) Build dir on the SSD RAID + various bits of tuning.
> 
> The results are basically the same, so it seems as if the SSD RAID
makes no difference. Benchmarking it does show twice the read/write
performance of the OS SSD, as expected. Disabling journalling and
increasing the commit time to 6000 also made no significant difference
to the build times, which were (to the nearest minute):


That is not surprising. With 4 cores and a very serialized build target,
I would not expect your SSD to be the bottleneck.

> 
> Real   : 42m
> User   : 133m
> System : 19m
> 
> These time were starting from nothing, and seem to fit with your 30
minutes with 3 times as many cores! BTW, BB_NUMBER_THREADS was set to 16
and PARALLEL_MAKE to 12.

A couple of things to keep in mind here. The minimal build is very
serialized in comparison to something like a sato build. If you want to
optimize your build times, look at the bbmatrix* scripts shipped with
poky to find the sweet spot for your target image and your build system.
I suspect you will find your BB_NUMBER_THREADS and PARALLEL_MAKE
settings are two high for your system. I'd start with them at 8 and 8,
or 8 and 6 respectively.

> 
> I also tried rebuilding the kernel:
>    bitbake -c clean linux-yocto
>    rm -rf the sstate bits for the above
>    bitbake linux-yocto
> 
> and got the following times:
> 
> Real   : 39m
> User   : 105m
> System : 16m
> 
> Which kind of fits with an observation. The minimal build had
something like 1530 stages to complete. The first 750 to 800 of these
flew past with all 8 'cores' running at just about 100% all the time.
Load average (short term) was about 19, so plenty ready to run. However,
round about the time python-native, the kernel, libxslt, gettext kicked
in the cpu usage dropped right off - to the point that the short term
load average dropped below 3. It did pick up again later on (after the
kernel was completed) before slowing down again towards the end (when it
would seem reasonable to expect that less can run in parallel).
> 
> It seems as if some of these bits (or others around this time)
> aren't
making use of parallel make or there is a queue of dependent tasks that
needs to be serialized.
> 
> The kernel build is a much bigger part of the build than I was
expecting, but this is only a small image. However, it looks as if the
main compilation phase completes very early on and a lot of time is then
spent building the modules (in a single thread, it seems) and in
packaging - which leads me to ask if RPM is the best option (speed
wise)? I don't use the packages myself (though understand they are
needed internally), so I can use the fastest (if there is one).

IPK is faster than RPM. This is what I use on most of my builds.

> 
> Is there anything else I should be considering to improve build
> times?

Run the ubuntu server kernel to eliminate some scheduling overhead.
Reducing the parallel settings mentioned above should help here too.

Welcome to Ubuntu 11.10 (GNU/Linux 3.0.0-16-server x86_64)

dvhart at rage:~
$ uname -r
3.0.0-16-server


As I said above, this is just a rough-cut at some benchmarking and I
plan to do some more, especially if there are other things to try and/or
any other information that would be useful.
> 
> Still, it's looking much, much faster than my old build system :-)
> 
> Chris Tapp
> 
> opensource at keylevel.com
> www.keylevel.com

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel