[yocto] Framework to implement mirroring in Yocto

[Andre McCurdy]

On Fri, Oct 6, 2017 at 3:08 PM, Gutierrez, Hernan Ildefonso (Boise
R&D, FW) <hernan_gutierrez at hp.com> wrote:
> Hi,
>
> We are planning to implement a mirror for both source code downloaded and
> sscache in our work environment.
>
> We are planning to use Nexus and Nuget to allow storage and versioning
> control. We don’t know if these are the right tools.

There are two main aspects of a mirror, fetching from the mirror
(happens during the build process) and populating the mirror
(generally happens outside the main build process).

For fetching from the mirror during the build process, the bitbake
fetchers are used, so your server needs to be accessible via a
protocol which the fetchers support, e.g http or https would be
recommended. Support for sstate mirrors via https with a password (ie
if you want private mirrors which can be accessed over the internet
without needing a VPN) was first added in pyro, so you'll need to
backport a couple of patches to make that work with morty. If the
standard bitbake fetchers don't meet your needs then it's not a huge
task to write a custom fetcher - see the Amazon S3 fetcher recently
added to bitbake. It's less than 100 lines of code.

For populating the mirror, you will likely need to implement something
custom, ie outside bitbake and the build process, since bitbake only
really supports writing sstate and downloads to local directories. It
could be something as simple as running rsync after every build to
upload from the build server's local sstate and downloads directories
to the mirror server(s). It could be something more complex - it's up
to you.

Note that for both sstate and downloads the files on the mirror would
be expected to be unique (e.g. there would never be two different
versions of "gcc-6.3.0.tar.bz2", etc), so there's no obvious reason to
be able to snapshot or version the state of the mirrors. Just let the
files accumulate over time - the old versions won't be lost or
over-written.

> Since we are about to embark in this project, before starting I wanted to
> know if you have some pointers on how can be implemented a mirroring
> framework for yocto.
>
> Not sure if there are some common tools (other tools) with which yocto
> integrates nicely.
>
> Any pointers will be appreciated,
>