[poky] Chasing fetcher issues in circles

[Richard Purdie]

Hi Bruce,

I've been chasing a fetcher issue around in circles, I have something
concrete to report back now.

We need to have the fetcher code work for the kernel recipes without
network access. The problem is that the kernel git recipe is complex,
has multiple branches and supports a variety of configurations. The root
of some of the problems is the chunk of anonymous python in
kernel-yocto.bbclass.

The trouble is we have a race between the variable PV being expanded
*anywhere* and that anonymous python running. The race is that other
anonymous python may run before the recipes anonymous python and that
other code can touch trigger an expansion of PV.

What happens if PV is expanded first?

Basically, the fetcher variables aren't setup and it falls back to the
UNDEFINED machine, the revisions are then set based on the undefined
machine rather that the true machine branch and confusion ensues. Worse
again, the fetcher can cache the first set of values it calculates.

One symptom of this and how it triggers a network access is that
SRCREV_machine can be set to "yocto/standard/base". I added:

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=32dbc70e86c3b186658dcaa84a8c7cd76a50e069

a short while ago so at least this is something resolvable (the fetcher
would do a "git ls-remote" on this) but it still means a network access
and it still means the fetcher config is broken at some point and at
risk of being cached.

So firstly, to fulfil our network access concerns when the kernel is
used with an undefined machine, we need one more SRCREV to be added to
poky-default-revisions.inc. I've done this in the following rfc commit:

http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/kernelchange&id=3e7e2a6d2c144a4f45e3cc37a75f78285987d1f9


That is a valid fix we need regardless but I think this race window is
what has been causing various people (like Darren and Tom) a lot of pain
and I really want to stop that *ever* happening again.

Even with the above change, it just stops us hitting the network, the
race still happens and the data is inconsistent.

So the takeaway is that the anonymous python isn't working. As far as I
can tell we can do one of two things:

a) If we really need some python code, there is a nasty way we can hack
it in and ensure it runs before SRCPV is expanded. This is hacktastic
but the patch is:

http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/kernelchange&id=8c2105e563e1aa2271201b27242b2936b265c72d

and I've confirmed this does actually work as intended.

b) The other option is we reduce the amount of anonymous python and use
other means to set the require variables.

For example, the 2.6.34 bits could be turned into a separate class which
overrides the default options and is only included by the 2.6.34
recipe(s). For the branch manipulations performed in code it does look a
little backwards since it looks like KMACHINE is following the old style
kernel still. Shouldn't we be detecting the old style and converting
that to new in the 2.6.34 case rather than converting existing variables
to the .37 style?

My worry here is that manipulating variables behind the user's back
tends to confuse them so its fine for upgrading to a new version but
doing it for our new default version is a little concerning.

Anyhow, there is some food for thought here. The patches above did turn
out a lot nicer looking than I thought they would when I started writing
the email!

Cheers,

Richard