[poky] Chasing fetcher issues in circles

[Bruce Ashfield]

On Tue, Feb 8, 2011 at 5:49 AM, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> Hi Bruce,
>
> I've been chasing a fetcher issue around in circles, I have something
> concrete to report back now.
>
> We need to have the fetcher code work for the kernel recipes without
> network access. The problem is that the kernel git recipe is complex,
> has multiple branches and supports a variety of configurations. The root
> of some of the problems is the chunk of anonymous python in
> kernel-yocto.bbclass.
>
> The trouble is we have a race between the variable PV being expanded
> *anywhere* and that anonymous python running. The race is that other
> anonymous python may run before the recipes anonymous python and that
> other code can touch trigger an expansion of PV.
>
> What happens if PV is expanded first?
>
> Basically, the fetcher variables aren't setup and it falls back to the
> UNDEFINED machine, the revisions are then set based on the undefined
> machine rather that the true machine branch and confusion ensues. Worse
> again, the fetcher can cache the first set of values it calculates.
>
> One symptom of this and how it triggers a network access is that
> SRCREV_machine can be set to "yocto/standard/base". I added:
>
> http://git.pokylinux.org/cgit.cgi/poky/commit/?id=32dbc70e86c3b186658dcaa84a8c7cd76a50e069
>
> a short while ago so at least this is something resolvable (the fetcher
> would do a "git ls-remote" on this) but it still means a network access
> and it still means the fetcher config is broken at some point and at
> risk of being cached.
>
> So firstly, to fulfil our network access concerns when the kernel is
> used with an undefined machine, we need one more SRCREV to be added to
> poky-default-revisions.inc. I've done this in the following rfc commit:
>
> http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/kernelchange&id=3e7e2a6d2c144a4f45e3cc37a75f78285987d1f9
>
>
> That is a valid fix we need regardless but I think this race window is
> what has been causing various people (like Darren and Tom) a lot of pain
> and I really want to stop that *ever* happening again.
>
> Even with the above change, it just stops us hitting the network, the
> race still happens and the data is inconsistent.
>
> So the takeaway is that the anonymous python isn't working. As far as I
> can tell we can do one of two things:
>
> a) If we really need some python code, there is a nasty way we can hack
> it in and ensure it runs before SRCPV is expanded. This is hacktastic
> but the patch is:
>
> http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/kernelchange&id=8c2105e563e1aa2271201b27242b2936b265c72d
>
> and I've confirmed this does actually work as intended.
>
> b) The other option is we reduce the amount of anonymous python and use
> other means to set the require variables.
>
> For example, the 2.6.34 bits could be turned into a separate class which
> overrides the default options and is only included by the 2.6.34
> recipe(s). For the branch manipulations performed in code it does look a
> little backwards since it looks like KMACHINE is following the old style
> kernel still. Shouldn't we be detecting the old style and converting
> that to new in the 2.6.34 case rather than converting existing variables
> to the .37 style?
>
> My worry here is that manipulating variables behind the user's back
> tends to confuse them so its fine for upgrading to a new version but
> doing it for our new default version is a little concerning.

I already have a complete re-factored set of kernel recipes here,
I was holding them back since I wasn't sure about introducing
churn in the 1.0 end of cycle. Sounds like I should dust them
off.

The anonymous python exists for one reason only:

  - to keep the fetcher happy, in particular when you try and build
    a new board.

Now that the fetcher has changed, the complexity can go. I'll try
and spin some re-worked classes shortly.

Cheers,

Bruce

>
> Anyhow, there is some food for thought here. The patches above did turn
> out a lot nicer looking than I thought they would when I started writing
> the email!
>
> Cheers,
>
> Richard
>
>
>
> _______________________________________________
> poky mailing list
> poky at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"