[yocto] PREMIRROR

Thu Jul 25 16:48:18 PDT 2019

Just tried the externalsrc feature.  Works perfectly.  Exactly what I was
looking for.  Thanks so much, Rudolf!

--Russ

On Thu, Jul 25, 2019 at 4:59 PM Rudolf J Streif <rudolf.streif at ibeeto.com>
wrote:

> Inlining below.
> On 7/25/19 8:14 AM, Russell Peterson wrote:
>
> I think I have a somewhat better understanding of what is going on.
>
> First off, I was confused by the fact that the original error message I
> saw from do_unpack referenced the file (URL) at
> DL_DIR/git2/original_github_url.  What I didn't understand at the time was
> that while that file existed,* it was actually a link* to
> DL_DIR/git2/local_url.  So, do_unpack wasn't looking at the wrong bare
> repository as I thought.  It was unhappy because it didn't see the git
> commit in the local repo that it was looking for.  I use AUTOREV for
> SRCREV.  Apparently, even with PREMIRROR set and matched, bitbake will
> still fetch the git HEAD commit hash from the original URL in the recipe.
> The local git repo doesn't contain that commit.  When I sync the local git
> repo to the github repo things work as expected.
>
> Correct. If the source repo or the correct commit cannot be found from the
> PREMIRROR, bitbake will use SRC_URI from the recipe. You can block that
> behavior by setting BB_NO_NETWORK = "1" in which case bitbake will not be
> able to connect to any remote server. However, that does not help if your
> PREMIRROR is a network server. In this case use BB_ALLOWED_HOSTS to allow
> access to only the servers you want.
>
>
> Here is my problem.  What I am ultimately trying to do here is have 2 git
> repos.  1 public, 1 private.  For internal development we use the private
> git repo (a local gerrit server).  Just before we ship we update the public
> github repo.  The recipe always points to the github (public) repo so we
> don't need to mess with that before a release.  This way we can do nightly
> builds (I'm just talking nightly builds done by Jenkins... not developers
> using eSDK and devtool) and test.  Customers using yocto will be given a
> recipe that points to github and, importantly, those customers that do NOT
> use yocto simply fetch the project from github and manually build it in
> their own environment with their own tools.
>
> That is a common problem. I do this all the time. To do this add to your
> conf/local.conf file:
>
> INHERIT += "externalsrc"
> EXTERNALSRC_pn-myrecipe = "/path/to/source/tree"
>
> Alternatively, you could use a bbappend file for you recipe in a
> development layer, myrecipe.bbappend:
>
> inherit externalsrc
> EXTERNALSRC = "/path/to/source/tree"
>
> Now, that is exactly what devtool does for you but you can of course do it
> manually.
>
> :rjs
>
>
>
> I can't be the only one doing this.  Is there a best practices here
> (private vs. public repo)?
>
> --Russ
>
>
> On Wed, Jul 24, 2019 at 3:55 PM Rudolf J Streif <rudolf.streif at ibeeto.com>
> wrote:
>
>> Russell,
>>
>> That is exactly what devtool and the externalsrc class do. PREMIRROR is
>> the wrong approach for that.
>>
>> :rjs
>> On 7/24/19 12:53 PM, Russell Peterson wrote:
>>
>> Hi, Rudolf.
>>
>> I apologize for not being clear.  The idea here is that my recipe points
>> to github while, for my local development environment, I set a premirror to
>> match a specific github repository and translate it to a local directory.
>> That works.  The fetch matches the PREMIRROR and places a copy of the LOCAL
>> git repo in my DL_DIR.  The problem is, the do_unpack task is looking for
>> the git repo in the DL_DIR (git2/etc...)... but it's looking for the git
>> repo from github, not my local repo that the fetch task just put in DL_DIR.
>>
>> There are numerous reasons I'm doing this including the fact that I
>> cannot put a reference to my local repo in the bb file since I ship that
>> recipe to my customers and a local pathname is meaningless to them.  I
>> prefer to simply modify my local.conf with a PREMIRROR value that has a
>> specific regular expression that matches this particular github repo (and
>> hence does NOT effect all recipes).  This is why I wanted to use the
>> PREMIRROR function.  Unfortunately, it does not appear to work or at least
>> not work as I expected.
>>
>> --Russ
>>
>>
>>
>>
>> On Wed, Jul 24, 2019 at 2:57 PM Rudolf J Streif <rudolf.streif at ibeeto.com>
>> wrote:
>>
>>> Hi Russell,
>>>
>>> devtool and eSDK are different things. The purpose of PREMIRRORS is to
>>> set a mirror for all recipes. It's a way for organizations to control where
>>> their YP builds download sources from. It's not intended to be used for a
>>> single recipe. There is no need for that. You simply set SRC_URI in your
>>> recipe to your local git repo. That is what devtool does after downloading
>>> the sources from a remote repo. If you already have the remote repo cloned
>>> locally you can just point devtool to it.
>>>
>>> You can do it manually by creating your recipe and setting SRC_URI like
>>> this:
>>>
>>> SRC_URI = "git:///local/path/${PN};protocol=file"
>>>
>>> SRCREV = "${AUTOREV}"
>>>
>>> S = "${WORKDIR}/git"
>>>
>>> PREMIRRORS is only relevant for do_fetch not for do_unpack.
>>>
>>> :rjs
>>>
>>>
>>> On 7/24/19 11:28 AM, Russell Peterson wrote:
>>>
>>> Hi, Rudolf.
>>>
>>> Thanks for the reply.  Yes, I am aware of the eSDK functionality,
>>> however, I have some unique requirements that I am trying to work around.
>>> Regardless... what I am doing should work, no?  I simply want to use a
>>> local git repo (the directory itself hence protocol=file) instead of what
>>> the recipe specifies.  Looks like the fetch is working but the do_unpack
>>> task is ignoring PREMIRRORS (or at least the localpath variable seems
>>> wrong).
>>>
>>> --Russ
>>>
>>>
>>> On Wed, Jul 24, 2019 at 12:19 PM Rudolf J Streif <
>>> rudolf.streif at ibeeto.com> wrote:
>>>
>>>> Russell,
>>>>
>>>> You don't need PREMIRROR for this functionality. It's not exactly
>>>> intended for that use.
>>>>
>>>> The simplest way to achieve what you are looking for is to use devtool.
>>>> If I understand you correctly you are downloading sources from a remote
>>>> repo on GitHub but want to have them locally to make modifications? If so
>>>> use:
>>>>
>>>> devtool add myrecipe localsrc fetchuri
>>>>
>>>> from your build environment (you have to source oe-init-build-env
>>>> first). devtool then fetches the source from fetchuri into a directory
>>>> localsrc as a git repo and automatically creates the recipe for it.
>>>>
>>>> :rjs
>>>>
>>>>
>>>> On 7/23/19 1:49 PM, Russell Peterson wrote:
>>>>
>>>> Hello,
>>>>
>>>> I am looking to have bitbake pick up files for a particular recipe from
>>>> a local git repository using the PREMIRROR functionality.
>>>>
>>>> Basically, the recipe (bb file) points to github but in my local build
>>>> I add PREMIRROR_prepend = "git://.*/.*
>>>> git:///local/path/BASENAME;protocol=file\n"
>>>>
>>>> I will probably make the git regular expression more exact for my
>>>> specific github repo but this works for now.
>>>>
>>>> This all works (as I deleted the github download from the local
>>>> download directory) because I can see in the do_fetch log and the correct
>>>> (local) repo was found and placed in the DL_DIR.
>>>>
>>>> Problem is, do_unpack fails because it appears to be looking for the
>>>> original (github) SRC_URI.  Then it complains about "no up to date source
>>>> found: clone or directory not available or not up to date (shallow clone
>>>> not enabled)"
>>>>
>>>> Any help on what I am missing would be appreciated.
>>>>
>>>> Regards,
>>>>
>>>> Russell
>>>>
>>>>
>>>>
>>>> --
>>>> -----
>>>> Rudolf J Streif
>>>> CEO/CTO ibeeto
>>>> +1.855.442.3396 x700
>>>>
>>>> --
>>> -----
>>> Rudolf J Streif
>>> CEO/CTO ibeeto
>>> +1.855.442.3396 x700
>>>
>>> --
>> -----
>> Rudolf J Streif
>> CEO/CTO ibeeto
>> +1.855.442.3396 x700
>>
>> --
> -----
> Rudolf J Streif
> CEO/CTO ibeeto
> +1.855.442.3396 x700
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190725/afda38b8/attachment-0001.html>