[yocto] Downloading Git Archives

[Chuck Wolber]

Thanks for replying!

That is effectively what we have to do with the VM clone of the build
machine. We spin up the VM, do the build, see what is new in the downloads
directory, do our due diligence, and push the new stuff into the local
mirror behind the firewall. The primary build server must remain "sterile",
so everything has to be "hoisted" over the fence manually.

[Side Note: We also maintain a local clone of the openembedded and poky
repositories. Our build pins to a specific hash, which gets advanced after
we do our due diligence. This is when we discover that certain packages
need to be added to our ${LOCAL_MIRROR}. My desire is to automate as much
as possible, while exposing as little as possible.]

Whenever the nightly build fails due to a missing package, and I see a URL
with a (for example) .tar.gz in the build error log, I know my job is a lot
easier. I can simply wget from any Internet connected host, instead of
spinning up the VM clone, waiting for a build to complete, and then
figuring out what is new in the downloads directory.

I was hoping that there was an equally simple way of grabbing git
repository packages from an Internet connected host without having a
bitbake build cache them for me.

I understand that these packages are effectively a bare clone, but there
seems to be some additional non-obvious (to me at least) "magic" that makes
it more complicated than doing a git repo bare clone and wrapping it up
into a .tar.gz.

I am happy to start trawling through source code and figure it out on my
own, but I figured this had to be common knowledge to someone on this list.

..Ch:W..

On Thu, Sep 26, 2019 at 8:50 PM Rudolf J Streif <rudolf.streif at ibeeto.com>
wrote:

> Chuck,
>
> Unless I am missing something (which I cannot entirely rule out), the
> default PREMIRRORS should actually do what you want. For the poky distro
> configuration has this:
>
> PREMIRRORS ??= "\
> bzr://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
> cvs://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
> git://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
> gitsm://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
> hg://.*/.*    http://downloads.yoctoproject.org/mirror/sources/ \n \
> osc://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
> p4://.*/.*    http://downloads.yoctoproject.org/mirror/sources/ \n \
> svn://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n"
>
> This causes the fetcher to intercept and translate all the SCM protocols
> into http downloads from the YP mirror. Since you are saying wget works for
> you this should actually automatically work for you. If you build system
> can at least download using http then
>
> bitbake <target> --runall=fetch
>
> should download all the sources for you. Then you can disconnect the box
> from the Internet and start the build offline.
>
> :rjs
> On 9/26/19 8:06 PM, Chuck Wolber wrote:
>
> Hi All,
>
> Our build environment is stuck behind a firewall with no access to the
> Internet. When a Yocto build requires a new package, we normally extract
> the URL from the failed build log and "wget" the package from the upstream
> source, do our due diligence (checksums, etc), and then add it to the
> server pointed to by ${LOCAL_MIRROR}.
>
> Obviously that procedure does not work so well when a git repository needs
> to be fetched. We have tried various methods, but nothing seems to work
> reliably. Our current method is an annoying hack involving a virtual
> machine clone of the build server.
>
> I have searched far and wide, and cannot find any sort of procedure. Can
> someone point me at a procedure for translating something like git://
> git.yoctoproject.org/pseudo into a file we can store in our
> ${LOCAL_MIRROR}?
>
> Thank you,
>
> ..Ch:W..
>
> --
> -----
> Rudolf J Streif
> CEO/CTO ibeeto
> +1.855.442.3386 x700
>
>

-- 
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one."  -Sam Harris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190926/145f4005/attachment.html>