[yocto] Mpich and yocto ?

[Randy MacLeod]

On 2017-07-26 08:08 PM, Riko Ho wrote:
> Dear Yocto Team Member,
> 
> The cluster is using Ubuntu.
> I haven't checked all the packages on the other computer. I will.
> 
> My goal is getting more hardrive space and compiling speed.
> 
> How can I relate bitbake with mpicc or other compilers needed by yocto 
> (arm-linux-gcc, gcc, etc)?

Do you mean that you want to do part of the build using each node in the 
cluster? That's possible now using distcc/icecream
(https://github.com/icecc/icecream ) but I haven't done it in years
as explained below.

Oh, there's a Stack Overflow entry for YP+distcc/icecream:
 
https://stackoverflow.com/questions/14472175/distributed-compile-with-bitbake

and distcc is mentioned in the docs:
    http://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html


Anyway, this just distributes the compilation so:
  - build management,
  - recipe download, unpack, patch, configuring and packaging,
  - linking, etc
are still done centrally.

There are plans to improve the bitbake task manager so that, with
the recipe specific sysroot work that has been done in 2.2, we'd be
able to farm out all of the stages of a package's build to individual
nodes in a cluster:
    https://bugzilla.yoctoproject.org/show_bug.cgi?id=10684
I don't know if that work is a priority for the fall 2.4 release.

Note that depending on your workload, inter-recipe dependencies and
the long time taken to run configure for autotools-based packages is
the bottleneck. Therefore, buying a high-end ( >= 16+16 core )
system with lots (64GB+) RAM and a good IO sub-system results
in core-image-minimal building quickly, certainly under 40 minutes.
    https://wiki.yoctoproject.org/wiki/Build_Performance

Also for repeated builds by one or more developers, sstate-cache is
a huge speed-up with some cost in scripts and people to manage it.


I've wondered if anyone has played with bitbake on an OS that is
known as a single-system image:
    https://en.wikipedia.org/wiki/Single_system_image
On these systems, jobs are transparently moved from the master-node
to idle nodes in the cluster by the operating system. That would be
an interesting academic exercise but it wouldn't be useful in
practice since such OSs are not generally used as far as I know at
least.

I look forward to hearing what your goals and plans are.

../Randy


> 
> I have tested a small code with mpicc and the cluster do the job.
> 
> Thanks for the attention.
> 
> On Jul 27, 2017 3:23 AM, "Randy MacLeod" <randy.macleod at windriver.com 
> <mailto:randy.macleod at windriver.com>> wrote:
>  >
>  > On 2017-07-26 02:09 AM, Riko Ho wrote:
>  >>
>  >> How can we do that ?
>  >>
>  >> bitbake in which node ? I don't understand ?
>  >>
>  >>
>  >> On 26/07/17 13:57, Josef Holzmayr wrote:
>  >>>
>  >>> Hi
>  >>>
>  >>> On 26.07.2017 05:12, Riko Ho wrote:
>  >>>>
>  >>>> Does anyone know on how to run poky on mpich cluster ?
>  >
>  >
>  > That's a rather ambiguous question.
>  >
>  > What OS/Distro is the mpich cluster running?
>  > Have you installed the packages that are required on the host:
>  >    see: "The Build Host Packages" here:
>  >
>  > 
> http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html
>  >
>  > What do you hope to achieve using bitbake?
>  >
>  >
>  >>>
>  >>> As far as I can see, MPICH is a userland library, basically. So it 
> would be the other way round, you could probably run a mpich application 
> on a number of nodes that run some OE/Poky thing.
>  >>>
>  >>> Greetz
>  >
>  >
>  > There is an mpich recipe here:
>  > http://layers.openembedded.org/layerindex/recipe/33348/
>  >
>  > but again, it's not clear what your ultimate goal is.
>  >
>  > ../Randy
>  >
>  >>
>  >> --
>  >> *
>  >>
>  >>
>  >> /*******/
>  >> Sent by Ubuntu LTS 16.04,
>  >> Kind regards,
>  >> Riko Ho
>  >> /*******/
>  >>
>  >> *
>  >>
>  >>
>  >
>  >
>  > --
>  > # Randy MacLeod. SMTS, Linux, Wind River
>  > Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, 
> Canada, K2K 2W5
> 


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, 
Canada, K2K 2W5