[yocto] [RFC PATCH 0/1] update.py: fetch repos parallelly

[Robert Yang]

On 01/03/2018 06:15 PM, Paul Eggleton wrote:
> Hi Robert,
> 
> On Wednesday, 3 January 2018 6:23:57 PM NZDT Robert Yang wrote:
>> On 12/27/2017 09:23 PM, Robert Yang wrote:
>>> We have 124 layers, the "update.py  -b <branch>" costs about 10 minutes to
>>> finish the update for one branch, and we need update several branches
>>> periodly, which is really a little slow. So I'd like to make it can run
>>> parally, here are some thoughts about it:
>>> 1) Make fetch can run parallelly, this is easy to do, and safe enough as
>>> the   patches does
>>> 2) Make recipeparse can parse layers parallelly, we may need split
>>>      update_layer.py into two parts:
>>>      - The one which only does recipeparse, this costs a lot of time, and
>>>      can be  parallell.
>>>      - The one which writes to database (can't be parallel, and doesn't
>>>      have to)
>>>
>>
>> I found that there is an easy way to improve the performance, just use
>> update_layer.py as a module rather than an program can save a lot of time,
>> we can use multiprocessing.Process() to call the function to put it in a
>> subprocess, and remove bb modules when switch branches since it may not
>> be compatible, please see next email for details, it can reduce time from
>> 9m20s to 1m43s for 124 layers when everything is up to date totally.
> 
> OK, but:
> 
> 1) Does this work across python 2 and 3?

Sorry, does it still needs support python2 ? My oldest bitbake only supported
python3, so I didn't test it with python 2 since bitbake would raise the error:

$ /usr/bin/python ./update.py -b <branch>

   File "/workspace1/lyang1/layers-fetched/bitbake/lib/bb/__init__.py", line 28, 
in <module>
     raise RuntimeError("Sorry, python 3.4.0 or later is required for this 
version of bitbake")

> 
> 2) Are you 100% sure that this clears out the memory of anything that happens
> to get loaded? One of the issues we used to see with the earlier single
> program structure was "corruption" of the python variable/module space; the
> current fully separate processes completely avoid that.

Yes, I'm sure, "del sys.modules[name]" can unload the modules when child
process is terminated.

What I met before was tinfoil can't be unloaded, then there would be errors
when switch from old branch (YP 2.2) to new one (YP 2.4), del sys.modules[name]
can fix the problem.

// Robert

> 
> Cheers,
> Paul
>