[yocto] Bitbake and task offloading onto multiple cloud-based servers
Mark Hatle
mark.hatle at windriver.com
Fri Jan 4 14:43:12 PST 2013
On 1/4/13 2:56 PM, Alex J Lennon wrote:
>
> Can anybody advise on whether bitbake currently supports offloading of
> build tasks onto multiple systems? Perhaps cloud based?
I see Chris Larson has already answer the main question here.. but I've got some
additional insight I can add.
> I'm thinking that it would be more efficient for me if I could bring up
> a number of Amazon EC2 servers (or similar) then have bitbake
> parallelise the build onto those servers to significantly reduce my
> build times?
In the past I've had people ask me questions about remote builds, parallel
building packages, etc. In the past when I've attempted to figure out if it
would be useful, the cost of the data transfer time always comes into play. How
long does it take to simply perform the task locally vs go to the remote
machine, perform the task, and get the data back. For a lot of the tasks that
OE does, the remote tasks will end up being slower on a reasonably modern i7 (or
similar Xeon) machine. I'm not sure using something like a cloud server would
end up helping due to this.
From experience I'd say that if anything you'd likely want to modify the
bitbake scheduler with some type of locality information. I.e. if you extract
the sources on a node, you most likely want to patch, configure, compile,
install, QA and package on that node as well.... but that also means a way to
establish node specific tmp/work space is needed for performance as well.
> I see bitbake supports a level of task parallelisation on a single box.
>
> Can parallelisation of build onto multiple systems be achieved?
All of the above are possible, but I suspect there is a lot of design and
refactoring required to come up with a system that is both efficient and fast.
> Is it something that should even be a goal?
It's not currently a goal. But it does get asked about every now and then. It
would be nice to have something more then anecdotal information about if it
would be a good idea or not. This is the type of thing that I think would make
an excellent research project. (Hey and if it makes things able to be built for
more people, faster, and cheaper great! If not, we'd have a more informed
response to questions like this.)
--Mark
> Cheers,
>
> Alex
>
> _______________________________________________
> yocto mailing list
> yocto at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
More information about the yocto
mailing list