[poky] [RFC 0/4] socketed based data connection

[Tian, Kevin]

>From: Qing He
>Sent: Wednesday, December 22, 2010 1:25 AM
>
>On Wed, 2010-12-22 at 00:49 +0800, He, Qing wrote:
>> This patch set enables remote data access in the exec'd task process.
>
>It's intended to be proof of concept to an alternative approach
>to work around the slowness introduced by exec bitbake-runtask.
>The patch is still WIP.
>
>http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

looks the change is not intrusive as I originally thought. :-)

>
>Thanks,
>Qing
>
>>
>> Some test data:
>> I've conducted a poky-image-minimal build on two different machines
>> with and without the patch, the result are as follow:
>>
>> 2 core weybridge: worker count=4, make -j4
>>   without patch
>>     real: 161min
>>     user: 237min
>>     sys:  56min
>>   with patch
>>     real: 138min
>>     user: 196min
>>     sys:  45min
>> 4 core nehalem: worker count=4, make -j4
>>   without patch
>>     real: 88min
>>     user: 218min
>>     sys:  37min
>>   with patch
>>     real: 83min
>>     user: 160min
>>     sys:  29min
>>
>> the user+sys has an improvement of around 20%~30%, however the
>> real runtime improvement is less significant. It's probably
>> because of the thread workload unbalance on multicore, or in another
>> word, some CPUs are not fully loaded during the build. e.g. when
>> do_rootfs_rpm is run, there is probably no other tasks running.
>>
>> Haven't done any test against single thread running, but it should
>> have a much more considerable impact.
>>
>>
>> Known problems:
>> 1. fetcher is not working at the moment, maybe bitbake needs an explicit
>> InheritFromOS(), but that produces some inconsistency in PATH in my previous
>> run
>> 2. do_rootfs_rpm has some problem in the first run
>>
>>
>> Next step and possible future usage:
>> 1. organize remote data to a separate class
>> 2. although we have reverted to fork for task running, a remote data mechanism
>> would work for a more separate build (e.g. distributed build)
>> 3. the added code in bitbake-runtask is to run necessary initializations for a
>> working environment, this is part of cooker. The current cooker seems too heavy
>> and may be broken down into several pieces according to functionalitiies
>>
>> Thanks,
>> Qing
>>
>>
>> Pull URL: git://git.pokylinux.org/poky-contrib.git
>>   Branch: qhe/datasock
>>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock
>>
>> Thanks,
>>     Qing He
>> ---
>>
>>
>> Qing He (4):
>>   python method fix
>>   sstate.bbclass: workaround
>>   data_smart: remote data
>>   socket based worker data
>>
>>  bitbake/bin/bitbake-runtask  |   80
>+++++++++++++++++++++++++++++++++--------
>>  bitbake/lib/bb/data_smart.py |   25 ++++++++++++-
>>  bitbake/lib/bb/methodpool.py |    6 +++
>>  bitbake/lib/bb/parse/ast.py  |    2 +-
>>  bitbake/lib/bb/runqueue.py   |   68
>++++++++++++++++++++++++++++++++----
>>  meta/classes/sstate.bbclass  |    2 +-
>>  6 files changed, 157 insertions(+), 26 deletions(-)
>>
>_______________________________________________
>poky mailing list
>poky at yoctoproject.org
>https://lists.yoctoproject.org/listinfo/poky