[yocto] [RFC PATCH 0/1] update.py: fetch repos parallelly

[Robert Yang]

Hi Paul,

On 12/27/2017 09:23 PM, Robert Yang wrote:
> Hi Paul,
> 
> We have 124 layers, the "update.py  -b <branch>" costs about 10 minutes to
> finish the update for one branch, and we need update several branches periodly,
> which is really a little slow. So I'd like to make it can run parally, here
> are some thoughts about it:
> 1) Make fetch can run parallelly, this is easy to do, and safe enough as the
>     patches does
> 2) Make recipeparse can parse layers parallelly, we may need split
>     update_layer.py into two parts:
>     - The one which only does recipeparse, this costs a lot of time, and can be
>       parallell.
>     - The one which writes to database (can't be parallel, and doesn't have to)
>

I found that there is an easy way to improve the performance, just use
update_layer.py as a module rather than an program can save a lot of time,
we can use multiprocessing.Process() to call the function to put it in a
subprocess, and remove bb modules when switch branches since it may not
be compatible, please see next email for details, it can reduce time from
9m20s to 1m43s for 124 layers when everything is up to date totally.

// Robert

> I will work on it if no objections.
> 
> // Robert
> 
> The following changes since commit fa0b01082fb0596959841e95af8c516ae3f267a7:
> 
>    admin: fix dependency display and search (2017-12-21 16:14:17 +1300)
> 
> are available in the git repository at:
> 
>    git://git.pokylinux.org/poky-contrib rbt/layerindex_parallel
>    http://git.pokylinux.org/cgit.cgi//log/?h=rbt/layerindex_parallel
> 
> Robert Yang (1):
>    update.py: fetch repos parallelly
> 
>   layerindex/update.py | 47 ++++++++++++++++++++++++++++++++++-------------
>   settings.py          |  2 ++
>   2 files changed, 36 insertions(+), 13 deletions(-)
>