[yocto] tar ball vs. git development questions

[James Abernathy]

On Jan 23, 2012, at 9:01 AM, Bruce Ashfield wrote:

> On 12-01-23 08:10 AM, Gary Thomas wrote:
>> On 2012-01-23 05:51, jfabernathy wrote:
>>> On 01/22/2012 08:12 PM, Gary Thomas wrote:
>>>> On 2012-01-22 13:19, James Abernathy wrote:
>>>>> I have used both git and the tarball methods of bitbaking projects,
>>>>> all of them derivatives of the examples in the Yocto documentation.
>>>>> I was having issues using the local clone of
>>>>> the Yocto kernel git repository this weekend. I had successfully
>>>>> done that before, but I was rebuilding the PC workstation, and
>>>>> getting everything setup and tested some of the
>>>>> meta-intel BSPs to make sure I had everything right. Cloning the
>>>>> linux-yocto-3.0 repository was successful, but the bakes against it
>>>>> failed. I made sure I had poky-extras setup
>>>>> right, but I still had problems. To isolate the problem, I changed
>>>>> to building with the tarballs and everything worked fine.
>>>>> 
>>>>> So that got me thinking what are the differences between the 2 methods:
>>>>> 
>>>>> * I assume that if I use the tarball method, bitbake, using the
>>>>> recipes, pulls down files from the online repositories and puts
>>>>> those files into the centralized local download
>>>>> directory ($DL_DIR), allowing reuse instead of re-downloading each
>>>>> time. The content downloaded for linux-yocto-3.0 is exactly what
>>>>> would be pulled from the local repository if
>>>>> I used a local clone of the git repository for linux-yocto-3.0.
>>>>> * If my assumption above is correct, if I'm not modifying the source
>>>>> code of the kernel (only changing config parameters), then once
>>>>> you've run at least one build with the
>>>>> tarball method, the $DL_DIR directory contains all the files you'll
>>>>> need to build any image with linux-yocto-3.0. So there is no need to
>>>>> have a local clone of the kernel
>>>>> repository for speeding up development. Am I right?
>>>>> * If I have a successful creation of a bare clone of
>>>>> linux-yocto-3.0.git, how could builds of Edison packages be failing?
>>>>> That makes me concerned about using git and successfully
>>>>> repeating builds of stable branches like Edison.
>>>> 
>>>> If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
>>>> then you'll get tarballs which hold the git repositories after
>>>> download. You can then reuse these (by sharing the DL_DIR or
>>>> using a local mirror). Does that help with the issue you're seeing?
>>>> 
>>> I'm not sure it does. I don't want poky to do more work. I have my
>>> download directory, $DL_DIR, outside my build directory so I can keep
>>> it run to run, as mentioned in the comments
>>> in local.conf. I was trying to understand 2 basic questions:
>>> 
>>> 1. what could be causing build failures using a freshly created bare
>>> clone yesterday vs. using the poky-edison-6.0 tarball. It would be
>>> scary if you could clone the linux-yocto-3.0
>>> successfully one day and have it be used in a build successfully, but
>>> clone it another day and it not work. I figured that bitbake/poky
>>> pulled the same information into DL_DIR
>>> regardless of whether you pulled from the bare clone locally or
>>> straight from the on-line repository.
>> 
>> What kind of errors? Some details might help understand what the problem
>> is.
>> 
>>> 2. I was trying to look at items to speed up build runs. I thought
>>> that if the downloads in DL_DIR were reused if they existed, it would
>>> speed up a run. It seems to. So the
>>> question is, after the first build, the files I need for
>>> linux-yocto-3.0 are in DL_DIR regardless of whether they came from the
>>> online repository or the local bare clone. right?
>> 
>> I think so, but I'm not 100% sure.
> 
> From my experience, this is true. Regardless of the SRCI_URI, the
> downloads directory contains everything that poky/bitbake need to
> do builds after the initial fetch has been performed. And that's
> where subsequent updates are pullled.
> 
> Cheers,
> 
> Bruce
> 
I'm going to retest all this and try some of these tricks when I have some time. If I get any more errors with the local repository, I'll post them.

Thanks,  Jim A

>> 
>> The way I have my system set up, I never download anything more
>> than once :-) I have a shared source repository which I point to
>> using MIRRORS and then all my builds are relative to that. If there
>> is new code, it gets downloaded and saved (as a tarball) in my sources
>> repository to be used by subsequent builds. To do this, I just have
>> these lines in <distro>.conf
>> # Provide pre-staged sources
>> SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
>> INHERIT += "own-mirrors"
>> BB_GENERATE_MIRROR_TARBALLS = "1"
>> I don't define DL_DIR in local.conf so files only come from my mirror.
>> 
>> This process is so successful that I almost always run with
>> BB_NO_NETWORK="1"
>> which causes the fetcher to die if it can't find the file locally. If I
>> find
>> that I do need to download something new, I disable this and let the
>> fetcher
>> download it. I'll then have a saved tarball (either downloaded directly or
>> synthesized) in <build_dir>/downloads that I can push to my mirror.
>> 
>