[linux-yocto] looking for doc for in-house linux-kernel dev tree

[Bruce Ashfield]

On 13-06-12 11:27 AM, Jean-François Dagenais wrote:
> Hi Bruce, thanks for the answers!
>
> On 2013-06-10, at 1:04 AM, Bruce Ashfield wrote:
>
>> [...]
>>> Couple of questions:
>>> - What exactly does AUTOREV do?
>>
>> The fetcher always gets the latest commit on the branches. During
>> development, we want to pick up everything that is pushed, without
>> needing to tweak all the SRCREVs.
>>
>>> - Is it possible to get this bbappend to use the current working directory state (including un-committed changes)
>>>    using the SRC_URI and AUTOREV?
>>
>> The SRCREV will only control what the fetcher pulls in, and then
>> what is checked out. The compile phase will then start. If there
>> are patches, or other touches to files that aren't committed, they
>> will be built. But if you have uncommitted changes, you'll have
>> potential issues with branch switches, etc.
>
> You mean if I make changes in the non-bare repo clone found in
> tmp/work/.../linux-yoctoXYZ/linux after the bitbake build has set it up?

Not quite. SRCREV is only used when the fetcher is pulling from
the upstream source. Wherever SRC_URI is pointing. If you modify files
in your build directory, they'll be compiled. Assuming that you don't
clean and rebuild of course.

>
>>
>>> - what is the _pn_ and why is it absent in the poky-contrib example? (SRCREV_machine_${MACHINE}="${AUTOREV}" and
>>>    SRCREV_meta="${AUTOREV}")
>>
>> A bitbake OVERRIDE value. _pn_ "package name". You'll find other
>> examples floating around the tree, and various layers.
>
> I have found other examples yes, still don't know what it overrides, and why the
> linux-yocto_3.x.bbappends use that, but the poky-contrib
> linux-yocto_3.x.bbappends don't.

It should be in the oe-core/yocto manuals, but I'd describe it as a more
'specific' variable. What you can use as override values is defined by
the OVERRIDES variable. <var>_OVERRIDE = will be used over simply <var> =.

_pn_ means that only when the package being built matches the name, will the
override take place. So it is specific to a recipe.

>
>
>>
>>> - in SRC_URI what does the nocheckout=1 and bareclone=[0-1] do? I've had error reports about this in
>>>    tmp/work/machine/linux-yoctoXYZ/temp logs.
>>
>> The tools work on the branches, to quickly make a copy of the tree
>> that has all the branches local, a bareclone and checkout in the
>> build directory is done. That is what those options control.
>
> Right... Found the most detailed doc I could at:
> poky/bitbake/lib/bb/fetch2/git.py Is the term bareclone the same as in git
> terminology? I.e. a bare clone is just the .git directory, no files are checked
> out of the git DB. I could not detect any difference between "bareclone=1" and
> "nocheckout=1", nor could I find any "git bare clones" in tmp/work. The
> downloads directory contains a true "git bare clone" of the repo mentioned in
> SRC_URI in either cases (nocheckout=1 or bareclone=1). Still a bit fuzzy, I am
> gathering that this is more of an optimization than anything perhaps?

It was required in the early days, now it is more of an optimization. You'd
only see the bare clone in your tmp/dowloads/* directory structure, not
in tmp/work, since the linux-yocto* recipes and classes debare the
tree and make it ready for building.

>
>
>>
>>>
>>> [...]
>>> We did some more experiments and found a lot of confusion around recipe-space's
>>> "KBRANCH_osprey = x" vs. meta-space's "branch osprey" statements. I have
>>> observed that if in my gitlab repo, I create a branch called "standard/osprey"
>>> and this branch is a merge of my-master + emgd-1.16 + standard/crownbay, the
>>> recipe build tries to patch things like kprobes and such, and basically fails
>>> saying:
>>>
>>> | ERROR: branch standard/osprey was requested, but was not properly
>>> |        configured to be built. The current branch is standard/osprey/base
>>>
>>> 'standard/osprey/base' has not be mentionned anywhere that I can see, it's a
>>> product of the recipe's work.
>>
>> It's a property of there being a subbranch of "standard/osprey" being
>> detected. Due to the way that git stores refs on disk, we have to
>> rename "standard/osprey" to standard/osprey/base if something like
>> standard/osprey/bsp is being requested.
>
>>
>>>
>>> If I don't prefix with "standard/" in KBRANCH, then it doesn't patch and things
>>> seems to work correctly.  (BTW, my gitlab is a clone of your linux-yocto-3.4, we
>>> just add new branches to that)
>>>
>>> Is this behaviour normal? If so, is there documentation for this? Another detail
>>> when I had the prefix "standard/" in my setup: I have temporarily tried to set
>>> branch "standard/base" at the same sha1 as my standard/osprey branch, that made
>>> no difference.
>>
>> It is normal, and just the tools trying to help you out. The docs
>> may be slim in this area, I'll double check tomorrow.
>>
>>>
>>>>
>>>> What does your BSP's .scc file look like ? Is it something you
>>>> can share ?
>>> bsp/osprey/osprey.scc:
>>> define KMACHINE osprey
>>> define KTYPE standard
>>> define KARCH i386
>>>
>>> include ktypes/standard/standard.scc
>>> branch osprey
>>>
>>> kconf hardware bsp/common-sonatest/hardware.cfg
>>> kconf hardware bsp/common-sonatest/w1.cfg
>>> kconf hardware osprey.cfg
>>>
>>> kconf non-hardware bsp/common-sonatest/non-hardware.cfg
>>>
>>> include cfg/x86.scc
>>> include features/irq/irq.scc
>>> include features/intel-e1xxxx/intel-e100.scc
>>> include features/intel-e1xxxx/intel-e1xxxx.scc
>>> include cfg/dmaengine.scc
>>> include cfg/8250.scc
>>> include features/power/intel.scc
>>>
>>> include features/usb/ehci-hcd.scc
>>> include features/usb/ohci-hcd.scc
>>>
>>> include features/aufs/aufs.scc
>>>
>>> include features/drm-emgd/drm-emgd-1.16.scc
>>>
>>> # default policy for standard kernels
>>> include cfg/usb-mass-storage.scc
>>> include cfg/boot-live.scc
>>> include features/latencytop/latencytop.scc
>>> include features/profiling/profiling.scc
>>>
>>> aufs.scc looks like:
>>> kconf non-hardware aufs.cfg
>>> branch aufs
>>>
>>> and branch aufs exists in my linux clone. It's a merge of upstream-aufs-3.4 +
>>> v3.4.46 + standard/base. The resulting aufs branch is merged into my machine
>>> branch in my clone.
>>
>> aha. And that's what is triggering the problem. Without extra
>> instructions, branch names are "additive", that's how the hierarchy
>> gets built up without encoding the names all over the .scc files.
>>
>> So that "branch aufs" means, create:
>>
>>   standard/osprey/aufs
>>
>> .. and hence what I described above must come true.
>>
>> So you are looking for aufs, just like i have in the 3.8 tree already?
>> In that case, what you can do is simply create that branch and merge
>> your changes. i.e. without describing the branch in the .scc file.
>
> You mean make my aufs.scc a one-liner that's just "kconf non-hardware aufs.cfg"?
> But what about the other scc's that we include? if they mention feature names in
> using the scc "branch" statement, won't the tooling look for

The tools will indeed look for the branch. But most features are not
branched. They are simply included and stacked within the existing
branch. Only features that for some reason we want to track and merge
independently are "staged" and then merged. The patches and changes are
already tracked via the meta-data, so we don't need a branch per
feature, generally speaking.

> standard/osprey/featureX which doesn't exist but its code is already merged in
> the machine branch we mentioned in KBRANCH?
>
>
>>
>> If you've already merged it into your BSP branch, no need to merge
>> it again, but you could just make the aufs.scc be:
>>
>>   git merge aufs
> Sorry, maybe just a typo, but since I am a bit confused about all this, let's
> make sure: you said if I DON'T need to merge it again? If so, wouldn't the "git
> merge aufs" in aufs.scc tell the kernel tools to make git merge aufs... which is
> already merged in my machine branch? I guess I don't understand what the scc
> "branch xyz" vs. "git merge xyz" statements mean or do.

This is something that we are cleaning up the documentation for at the
moment, so it not being clear isn't surprising.

.scc files create a "meta-series', which is just like a quilt series,
but it contains information about patches, tree operations, configuration
and more. All in an in-order representation of what happens to build
a given BSP.

So in a .scc file, "branch foo", means from the current point, create
branch "foo", and then continue processing. Which means you'd have
something like:

   branch a
     patch 1
     patch 2
        branch foo
           patch 3
           patch 4

If you use the "branch" directive. They are just logical separations in
the features, which helps during maintenance phases.

"git merge bar" means, merge branch "bar" into the current branch,
merging the histories. You can think of it happening right at the
point where we are in the meta series, but git is merging the
histories, so anything on that branch is a potential conflict, not
only the patches that have just been applied.

There's no harm in multiple merges of the same feature, git will handle
it and just do nothing on the subsequent attempts.


>
>
>>
>> And your BSP will merge the changes into it's branch on each build.
>>
>
> In the end though, for this scenario to work as expected, we would need to have
> all the intelligent patching and merging completely disabled, while keeping the
> config fragment parts still working. While we seem to have stumbled on aWhat
> combination of flags, branch names and variable names should we use/avoid to
> obtain this desired behaviour?

You shouldn't need to disable anything in particular. There are options
in yocto 1.5 to inhibit patches or configs, but generally speaking, they are
all placed properly and accounted for, so they shouldn't cause a problem.

Cheers,

Bruce

>
> Thanks for your patience and help. I hope others will benefit from our pains!
> Cheers!
> /jfd
>
>