[yocto] [PATCH] [yocto-ab-helper] utils.py: Resolved unicode data expansion

[Chan, Aaron Chun Yew]

Hello Richard,

This morning I had the new autobuilder setup from scratch with the latest patch you just check out, thanks for that and rollout the fix below manually. Everything else is clean and I did what you asked me to.
However this did not resolved the data expansion when called utils.getconfig("REPO_STASH_DIR, ourconfig), this applies when you invoke ourconfig["REPO_STASH_DIR"]. Both yields the same errors.
We assume the  JSON dumps are properly handled in ourconfig[c] when we handles config[c] but that is not the case. I do see there is a growing issues as your strategy to use nested JSON, however 
we wont be able to handle all of these conditions needed when nested JSON becomes complex. Anyway, i'll leave it you decide what will be best course of action.

STDERR logs on autobuilder: (poky-tiny)
--------------------------------------
mv: cannot move '/home/pokybuild/yocto-worker/poky-tiny/' to a subdirectory of itself, '${BASE_HOMEDIR}/git/mirror/1530669213-56172/poky-tiny'
Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder-helper/janitor/clobberdir", line 52, in <module>
    subprocess.check_call(['mv', x, trashdest])
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mv', '/home/pokybuild/yocto-worker/poky-tiny/', u'${BASE_HOMEDIR}/git/mirror/1530669213-56172']' returned non-zero exit status 1

also, this action causes directory named ${BASE_HOMEDIR} to be created under ~/yocto-worker/poky-tiny/build

This patch [https://lists.yoctoproject.org/pipermail/yocto/2018-July/041685.html] which submitted today resolves the Step 1: Clobber build dir on autobuilder. 

Best wishes,
Aaron
________________________________________
From: richard.purdie at linuxfoundation.org [richard.purdie at linuxfoundation.org]
Sent: Tuesday, July 03, 2018 9:25 PM
To: Chan, Aaron Chun Yew; yocto at yoctoproject.org
Subject: Re: [PATCH] [yocto-ab-helper] utils.py: Resolved unicode data expansion

On Tue, 2018-07-03 at 09:44 +0800, Aaron Chan wrote:
> Updated patch to trigger handlestr() when unicode string is found
> during iteration json.loads(config.json). Unicode and list with data
> expansion were not handled hence adding this patch to handle
> conversion.
> Added a debug message to dump pretty json data populated to
> ourconfig[c].
>
> e.g "REPO_STASH_DIR" read as ${BASE_HOMEDIR}/git/mirror, where it
> should be
>     "REPO_STASH_DIR" as /home/pokybuild/git/mirror
>
> Signed-off-by: Aaron Chan <aaron.chun.yew.chan at intel.com>
> ---
>  scripts/utils.py | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)

It took me a while to figure out why you were doing this.

We can't expand the data half way through loading the json file as
other pieces of data may later override the values. We therefore have
to defer expansion of variables until the file is completely loaded.

We therefore have to expand the variables later on, when we read them.

I pointed you at this commit:

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/commit/?id=d6253df2bc21752bc0b53202e491140b0994ff63

which changes direct accesses into ourconfig, e.g.:

ourconfig["REPO_STASH_DIR"]

into accesses using a function:

utils.getconfig("REPO_STASH_DIR", ourconfig)

and that function handles the expansion.

You should therefore be able to fix the clobberdir issue by using the
getconfig() method instead of direct access?

Cheers,

Richard