[yocto] configure optimization feature update

[Xu, Dongxiao]

> -----Original Message-----
> From: Khem Raj [mailto:raj.khem at gmail.com]
> Sent: Thursday, June 16, 2011 9:29 AM
> To: Xu, Dongxiao
> Cc: Richard Purdie (richard.purdie at linuxfoundation.org);
> yocto at yoctoproject.org
> Subject: Re: [yocto] configure optimization feature update
> 
> On Wed, Jun 15, 2011 at 5:57 PM, Xu, Dongxiao <dongxiao.xu at intel.com>
> wrote:
> > Hi Richard,
> >
> > Recently I was doing the "configure optimization" feature and collecting data
> for it.
> >
> > The main logic of this feature is straight forward:
> >
> > 1. Use the diff file as autoreconf cache. (I use command: "diff -ruN
> SOURCE-ORIG SOURCE", here "SOURCE-ORIG" is the source directory before
> running autoreconf, while "SOURCE" is the directory after running autoreconf).
> > 2. Add SRC_URI checksum for all patches of the source code.
> > 3. Tag each autoreconf cache file with ${PN} and the SRC_URI checksum of
> source code and all patches.
> > 4. If the currently SRC_URI checksum matches the cached checksum, then we
> can patch the cache instead of running "autoreconf" stage.
> >
> 
> The autoconf'ing is sort of arbitrary at the moment. Depending on what is
> staged the results may vary. So some way of creating a cache is nice  since it
> can be used to verify if package rebuild happens with same configure variables
> or not. Sometimes we have seen some packages are build differently first time
> since many packages are not staged.
> 
> 
> > I did some testings for sato build, the result is not as good as we expected:
> >
> > On a server build machine (Genuine Intel(R) CPU @ 2.40GHz, 2 sockets with 6
> core each and hyperthreading, thus 24 logical CPUs in all, 66G memory):
> >
> > w/o the optimization:
> > real    83m40.963s
> > user    496m58.550s
> > sys     329m1.590s
> >
> > w/ the optimization:
> > real    79m1.062s
> > user    460m58.600s
> > sys     347m42.120s
> >
> > It has about 5% performance gain.
> >
> > I also tested the patch on a desktop core-i7 machine (Intel(R) Core(TM) i7
> CPU 870 @ 2.93GHz, 4 core 8 logical CPU, 4G memory):
> >
> > w/o the optimization:
> > real    105m25.436s
> > user    372m48.040s
> > sys     51m23.950s
> >
> > w/ the optimization:
> > real    103m38.314s
> > user    332m35.770s
> > sys     49m4.520s
> >
> > It only has about 2% performance gain.
> >
> > The result is not encouraging.
> >
> > There are also some other things we need to take into consideration for this
> feature:
> >
> > 1. If add this feature, the first build time should be longer than current since it
> needs to build the autoreconf cache.
> > 2. Maintainers needs to maintain the SRC_URI checksums not only for source
> code, but also all its patches. For some recipes, it has more than 20 patches,
> which needs assignable maintenance effort.
> 
> Yeah thats definite pain.
> 
> > 3. How to distribute the caches will be a problem. The total size of such cache
> is about 900M (before compression) and 200M (after compression). Since the
> size is not small, distributing it with Poky source code doesn't make sense. On
> another aspect, we can use something like "sstate". But since we already have
> caches of sstate, I think it is not necessary for us to enable another similar
> cache mechanism with little improvement.
> >
> 
> hmm this is a real problem and probably the perf killer. I wonder why the sizes
> are so big.

I diff the source files before and after autoreconf and using the generated patch as our autoreconf cache. For a certain source code, for example connman, such diff file will be as large as 3M, and for a sato build, we have about 300 recipes need autoreconf.

Thanks,
Dongxiao

> 
> > Therefore my opinion is we may give up this feature. What's your comments
> and suggestions?
> >
> > Thanks,
> > Dongxiao
> >
> > _______________________________________________
> > yocto mailing list
> > yocto at yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >