[yocto] PREMIRROR

[Russell Peterson]

I think I have a somewhat better understanding of what is going on.

First off, I was confused by the fact that the original error message I saw
from do_unpack referenced the file (URL) at
DL_DIR/git2/original_github_url.  What I didn't understand at the time was
that while that file existed,* it was actually a link* to
DL_DIR/git2/local_url.  So, do_unpack wasn't looking at the wrong bare
repository as I thought.  It was unhappy because it didn't see the git
commit in the local repo that it was looking for.  I use AUTOREV for
SRCREV.  Apparently, even with PREMIRROR set and matched, bitbake will
still fetch the git HEAD commit hash from the original URL in the recipe.
The local git repo doesn't contain that commit.  When I sync the local git
repo to the github repo things work as expected.

Here is my problem.  What I am ultimately trying to do here is have 2 git
repos.  1 public, 1 private.  For internal development we use the private
git repo (a local gerrit server).  Just before we ship we update the public
github repo.  The recipe always points to the github (public) repo so we
don't need to mess with that before a release.  This way we can do nightly
builds (I'm just talking nightly builds done by Jenkins... not developers
using eSDK and devtool) and test.  Customers using yocto will be given a
recipe that points to github and, importantly, those customers that do NOT
use yocto simply fetch the project from github and manually build it in
their own environment with their own tools.

I can't be the only one doing this.  Is there a best practices here
(private vs. public repo)?

--Russ


On Wed, Jul 24, 2019 at 3:55 PM Rudolf J Streif <rudolf.streif at ibeeto.com>
wrote:

> Russell,
>
> That is exactly what devtool and the externalsrc class do. PREMIRROR is
> the wrong approach for that.
>
> :rjs
> On 7/24/19 12:53 PM, Russell Peterson wrote:
>
> Hi, Rudolf.
>
> I apologize for not being clear.  The idea here is that my recipe points
> to github while, for my local development environment, I set a premirror to
> match a specific github repository and translate it to a local directory.
> That works.  The fetch matches the PREMIRROR and places a copy of the LOCAL
> git repo in my DL_DIR.  The problem is, the do_unpack task is looking for
> the git repo in the DL_DIR (git2/etc...)... but it's looking for the git
> repo from github, not my local repo that the fetch task just put in DL_DIR.
>
> There are numerous reasons I'm doing this including the fact that I cannot
> put a reference to my local repo in the bb file since I ship that recipe to
> my customers and a local pathname is meaningless to them.  I prefer to
> simply modify my local.conf with a PREMIRROR value that has a specific
> regular expression that matches this particular github repo (and hence does
> NOT effect all recipes).  This is why I wanted to use the PREMIRROR
> function.  Unfortunately, it does not appear to work or at least not work
> as I expected.
>
> --Russ
>
>
>
>
> On Wed, Jul 24, 2019 at 2:57 PM Rudolf J Streif <rudolf.streif at ibeeto.com>
> wrote:
>
>> Hi Russell,
>>
>> devtool and eSDK are different things. The purpose of PREMIRRORS is to
>> set a mirror for all recipes. It's a way for organizations to control where
>> their YP builds download sources from. It's not intended to be used for a
>> single recipe. There is no need for that. You simply set SRC_URI in your
>> recipe to your local git repo. That is what devtool does after downloading
>> the sources from a remote repo. If you already have the remote repo cloned
>> locally you can just point devtool to it.
>>
>> You can do it manually by creating your recipe and setting SRC_URI like
>> this:
>>
>> SRC_URI = "git:///local/path/${PN};protocol=file"
>>
>> SRCREV = "${AUTOREV}"
>>
>> S = "${WORKDIR}/git"
>>
>> PREMIRRORS is only relevant for do_fetch not for do_unpack.
>>
>> :rjs
>>
>>
>> On 7/24/19 11:28 AM, Russell Peterson wrote:
>>
>> Hi, Rudolf.
>>
>> Thanks for the reply.  Yes, I am aware of the eSDK functionality,
>> however, I have some unique requirements that I am trying to work around.
>> Regardless... what I am doing should work, no?  I simply want to use a
>> local git repo (the directory itself hence protocol=file) instead of what
>> the recipe specifies.  Looks like the fetch is working but the do_unpack
>> task is ignoring PREMIRRORS (or at least the localpath variable seems
>> wrong).
>>
>> --Russ
>>
>>
>> On Wed, Jul 24, 2019 at 12:19 PM Rudolf J Streif <
>> rudolf.streif at ibeeto.com> wrote:
>>
>>> Russell,
>>>
>>> You don't need PREMIRROR for this functionality. It's not exactly
>>> intended for that use.
>>>
>>> The simplest way to achieve what you are looking for is to use devtool.
>>> If I understand you correctly you are downloading sources from a remote
>>> repo on GitHub but want to have them locally to make modifications? If so
>>> use:
>>>
>>> devtool add myrecipe localsrc fetchuri
>>>
>>> from your build environment (you have to source oe-init-build-env
>>> first). devtool then fetches the source from fetchuri into a directory
>>> localsrc as a git repo and automatically creates the recipe for it.
>>>
>>> :rjs
>>>
>>>
>>> On 7/23/19 1:49 PM, Russell Peterson wrote:
>>>
>>> Hello,
>>>
>>> I am looking to have bitbake pick up files for a particular recipe from
>>> a local git repository using the PREMIRROR functionality.
>>>
>>> Basically, the recipe (bb file) points to github but in my local build I
>>> add PREMIRROR_prepend = "git://.*/.*
>>> git:///local/path/BASENAME;protocol=file\n"
>>>
>>> I will probably make the git regular expression more exact for my
>>> specific github repo but this works for now.
>>>
>>> This all works (as I deleted the github download from the local download
>>> directory) because I can see in the do_fetch log and the correct (local)
>>> repo was found and placed in the DL_DIR.
>>>
>>> Problem is, do_unpack fails because it appears to be looking for the
>>> original (github) SRC_URI.  Then it complains about "no up to date source
>>> found: clone or directory not available or not up to date (shallow clone
>>> not enabled)"
>>>
>>> Any help on what I am missing would be appreciated.
>>>
>>> Regards,
>>>
>>> Russell
>>>
>>>
>>>
>>> --
>>> -----
>>> Rudolf J Streif
>>> CEO/CTO ibeeto
>>> +1.855.442.3396 x700
>>>
>>> --
>> -----
>> Rudolf J Streif
>> CEO/CTO ibeeto
>> +1.855.442.3396 x700
>>
>> --
> -----
> Rudolf J Streif
> CEO/CTO ibeeto
> +1.855.442.3396 x700
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190725/514ddcaf/attachment.html>