[yocto] PREMIRROR
Rudolf J Streif
rudolf.streif at ibeeto.com
Fri Jul 26 15:19:29 PDT 2019
You are very welcome. Enjoy working with YP.
:rjs
On 7/25/19 4:48 PM, Russell Peterson wrote:
> Just tried the externalsrc feature. Works perfectly. Exactly what I
> was looking for. Thanks so much, Rudolf!
>
> --Russ
>
>
> On Thu, Jul 25, 2019 at 4:59 PM Rudolf J Streif
> <rudolf.streif at ibeeto.com <mailto:rudolf.streif at ibeeto.com>> wrote:
>
> Inlining below.
>
> On 7/25/19 8:14 AM, Russell Peterson wrote:
>> I think I have a somewhat better understanding of what is going on.
>>
>> First off, I was confused by the fact that the original error
>> message I saw from do_unpack referenced the file (URL) at
>> DL_DIR/git2/original_github_url. What I didn't understand at the
>> time was that while that file existed,/it was actually a link/ to
>> DL_DIR/git2/local_url. So, do_unpack wasn't looking at the wrong
>> bare repository as I thought. It was unhappy because it didn't
>> see the git commit in the local repo that it was looking for. I
>> use AUTOREV for SRCREV. Apparently, even with PREMIRROR set and
>> matched, bitbake will still fetch the git HEAD commit hash from
>> the original URL in the recipe. The local git repo doesn't
>> contain that commit. When I sync the local git repo to the
>> github repo things work as expected.
> Correct. If the source repo or the correct commit cannot be found
> from the PREMIRROR, bitbake will use SRC_URI from the recipe. You
> can block that behavior by setting BB_NO_NETWORK = "1" in which
> case bitbake will not be able to connect to any remote server.
> However, that does not help if your PREMIRROR is a network server.
> In this case use BB_ALLOWED_HOSTS to allow access to only the
> servers you want.
>>
>> Here is my problem. What I am ultimately trying to do here is
>> have 2 git repos. 1 public, 1 private. For internal development
>> we use the private git repo (a local gerrit server). Just before
>> we ship we update the public github repo. The recipe always
>> points to the github (public) repo so we don't need to mess with
>> that before a release. This way we can do nightly builds (I'm
>> just talking nightly builds done by Jenkins... not developers
>> using eSDK and devtool) and test. Customers using yocto will be
>> given a recipe that points to github and, importantly, those
>> customers that do NOT use yocto simply fetch the project from
>> github and manually build it in their own environment with their
>> own tools.
>>
> That is a common problem. I do this all the time. To do this add
> to your conf/local.conf file:
>
> INHERIT += "externalsrc"
> EXTERNALSRC_pn-myrecipe = "/path/to/source/tree"
>
> Alternatively, you could use a bbappend file for you recipe in a
> development layer, myrecipe.bbappend:
>
> inherit externalsrc
> EXTERNALSRC = "/path/to/source/tree"
>
> Now, that is exactly what devtool does for you but you can of
> course do it manually.
>
> :rjs
>
>
>
>> I can't be the only one doing this. Is there a best practices
>> here (private vs. public repo)?
>>
>> --Russ
>>
>>
>> On Wed, Jul 24, 2019 at 3:55 PM Rudolf J Streif
>> <rudolf.streif at ibeeto.com <mailto:rudolf.streif at ibeeto.com>> wrote:
>>
>> Russell,
>>
>> That is exactly what devtool and the externalsrc class do.
>> PREMIRROR is the wrong approach for that.
>>
>> :rjs
>>
>> On 7/24/19 12:53 PM, Russell Peterson wrote:
>>> Hi, Rudolf.
>>>
>>> I apologize for not being clear. The idea here is that my
>>> recipe points to github while, for my local development
>>> environment, I set a premirror to match a specific github
>>> repository and translate it to a local directory. That
>>> works. The fetch matches the PREMIRROR and places a copy of
>>> the LOCAL git repo in my DL_DIR. The problem is, the
>>> do_unpack task is looking for the git repo in the DL_DIR
>>> (git2/etc...)... but it's looking for the git repo from
>>> github, not my local repo that the fetch task just put in
>>> DL_DIR.
>>>
>>> There are numerous reasons I'm doing this including the fact
>>> that I cannot put a reference to my local repo in the bb
>>> file since I ship that recipe to my customers and a local
>>> pathname is meaningless to them. I prefer to simply modify
>>> my local.conf with a PREMIRROR value that has a specific
>>> regular expression that matches this particular github repo
>>> (and hence does NOT effect all recipes). This is why I
>>> wanted to use the PREMIRROR function. Unfortunately, it
>>> does not appear to work or at least not work as I expected.
>>>
>>> --Russ
>>>
>>>
>>>
>>>
>>> On Wed, Jul 24, 2019 at 2:57 PM Rudolf J Streif
>>> <rudolf.streif at ibeeto.com <mailto:rudolf.streif at ibeeto.com>>
>>> wrote:
>>>
>>> Hi Russell,
>>>
>>> devtool and eSDK are different things. The purpose of
>>> PREMIRRORS is to set a mirror for all recipes. It's a
>>> way for organizations to control where their YP builds
>>> download sources from. It's not intended to be used for
>>> a single recipe. There is no need for that. You simply
>>> set SRC_URI in your recipe to your local git repo. That
>>> is what devtool does after downloading the sources from
>>> a remote repo. If you already have the remote repo
>>> cloned locally you can just point devtool to it.
>>>
>>> You can do it manually by creating your recipe and
>>> setting SRC_URI like this:
>>>
>>> SRC_URI = "git:///local/path/${PN};protocol=file"
>>>
>>> SRCREV = "${AUTOREV}"
>>>
>>> S = "${WORKDIR}/git"
>>>
>>> PREMIRRORS is only relevant for do_fetch not for do_unpack.
>>>
>>> :rjs
>>>
>>>
>>> On 7/24/19 11:28 AM, Russell Peterson wrote:
>>>> Hi, Rudolf.
>>>>
>>>> Thanks for the reply. Yes, I am aware of the eSDK
>>>> functionality, however, I have some unique requirements
>>>> that I am trying to work around. Regardless... what I
>>>> am doing should work, no? I simply want to use a local
>>>> git repo (the directory itself hence protocol=file)
>>>> instead of what the recipe specifies. Looks like the
>>>> fetch is working but the do_unpack task is ignoring
>>>> PREMIRRORS (or at least the localpath variable seems
>>>> wrong).
>>>>
>>>> --Russ
>>>>
>>>>
>>>> On Wed, Jul 24, 2019 at 12:19 PM Rudolf J Streif
>>>> <rudolf.streif at ibeeto.com
>>>> <mailto:rudolf.streif at ibeeto.com>> wrote:
>>>>
>>>> Russell,
>>>>
>>>> You don't need PREMIRROR for this functionality.
>>>> It's not exactly intended for that use.
>>>>
>>>> The simplest way to achieve what you are looking
>>>> for is to use devtool. If I understand you
>>>> correctly you are downloading sources from a remote
>>>> repo on GitHub but want to have them locally to
>>>> make modifications? If so use:
>>>>
>>>> devtool add myrecipe localsrc fetchuri
>>>>
>>>> from your build environment (you have to source
>>>> oe-init-build-env first). devtool then fetches the
>>>> source from fetchuri into a directory localsrc as a
>>>> git repo and automatically creates the recipe for it.
>>>>
>>>> :rjs
>>>>
>>>>
>>>> On 7/23/19 1:49 PM, Russell Peterson wrote:
>>>>> Hello,
>>>>>
>>>>> I am looking to have bitbake pick up files for a
>>>>> particular recipe from a local git repository
>>>>> using the PREMIRROR functionality.
>>>>>
>>>>> Basically, the recipe (bb file) points to github
>>>>> but in my local build I add PREMIRROR_prepend =
>>>>> "git://.*/.*
>>>>> git:///local/path/BASENAME;protocol=file\n"
>>>>>
>>>>> I will probably make the git regular expression
>>>>> more exact for my specific github repo but this
>>>>> works for now.
>>>>>
>>>>> This all works (as I deleted the github download
>>>>> from the local download directory) because I can
>>>>> see in the do_fetch log and the correct (local)
>>>>> repo was found and placed in the DL_DIR.
>>>>>
>>>>> Problem is, do_unpack fails because it appears to
>>>>> be looking for the original (github) SRC_URI.
>>>>> Then it complains about "no up to date source
>>>>> found: clone or directory not available or not up
>>>>> to date (shallow clone not enabled)"
>>>>>
>>>>> Any help on what I am missing would be appreciated.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Russell
>>>>>
>>>>>
>>>>>
>>>> --
>>>> -----
>>>> Rudolf J Streif
>>>> CEO/CTO ibeeto
>>>> +1.855.442.3396 x700
>>>>
>>> --
>>> -----
>>> Rudolf J Streif
>>> CEO/CTO ibeeto
>>> +1.855.442.3396 x700
>>>
>> --
>> -----
>> Rudolf J Streif
>> CEO/CTO ibeeto
>> +1.855.442.3396 x700
>>
> --
> -----
> Rudolf J Streif
> CEO/CTO ibeeto
> +1.855.442.3396 x700
>
--
-----
Rudolf J Streif
CEO/CTO ibeeto
+1.855.442.3396 x700
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190726/25c1f461/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190726/25c1f461/attachment-0001.pgp>
More information about the yocto
mailing list