[poky] [RFC 0/4] socketed based data connection
Qing He
qing.he at intel.com
Tue Dec 21 08:49:04 PST 2010
This patch set enables remote data access in the exec'd task process.
Some test data:
I've conducted a poky-image-minimal build on two different machines
with and without the patch, the result are as follow:
2 core weybridge: worker count=4, make -j4
without patch
real: 161min
user: 237min
sys: 56min
with patch
real: 138min
user: 196min
sys: 45min
4 core nehalem: worker count=4, make -j4
without patch
real: 88min
user: 218min
sys: 37min
with patch
real: 83min
user: 160min
sys: 29min
the user+sys has an improvement of around 20%~30%, however the
real runtime improvement is less significant. It's probably
because of the thread workload unbalance on multicore, or in another
word, some CPUs are not fully loaded during the build. e.g. when
do_rootfs_rpm is run, there is probably no other tasks running.
Haven't done any test against single thread running, but it should
have a much more considerable impact.
Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicit
InheritFromOS(), but that produces some inconsistency in PATH in my previous
run
2. do_rootfs_rpm has some problem in the first run
Next step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism
would work for a more separate build (e.g. distributed build)
3. the added code in bitbake-runtask is to run necessary initializations for a
working environment, this is part of cooker. The current cooker seems too heavy
and may be broken down into several pieces according to functionalitiies
Thanks,
Qing
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: qhe/datasock
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock
Thanks,
Qing He
---
Qing He (4):
python method fix
sstate.bbclass: workaround
data_smart: remote data
socket based worker data
bitbake/bin/bitbake-runtask | 80 +++++++++++++++++++++++++++++++++--------
bitbake/lib/bb/data_smart.py | 25 ++++++++++++-
bitbake/lib/bb/methodpool.py | 6 +++
bitbake/lib/bb/parse/ast.py | 2 +-
bitbake/lib/bb/runqueue.py | 68 ++++++++++++++++++++++++++++++++----
meta/classes/sstate.bbclass | 2 +-
6 files changed, 157 insertions(+), 26 deletions(-)
More information about the poky
mailing list