[yocto] Framework to implement mirroring in Yocto

[Mark Hatle]

On 10/6/17 5:08 PM, Gutierrez, Hernan Ildefonso (Boise R&D, FW) wrote:
> Hi,
> 
>  
> 
> We are planning to implement a mirror for both source code downloaded and
> sscache in our work environment.
> 

We mirror a ton of code for our customers.  We have two types or mirrors that we
deliver.  Raw 'git servers' and tarball (or equivalent) mirrors.

The way that we populate these mirrors is to use bitbake with a fetchall
'universe'.  That way everything that could be downloaded (with a few
exceptions) is downloaded.

git archives, we identify and create git mirrors in an external server.  This
server regularly updates the git servers from the upstream location.  This
allows us to have git mirrors that are 'live' git servers, and not just tarballs
that could get out of sync.

For this work, you can easily setup a 'premirror' in your projects of the format:

PREMIRRORS_append = " \
     git://.*/.* git://mirror.hostname.here/git/MIRRORNAME;protocol=git \n \"
"

You can adjust the protocol to be file, git, or http (as necessary for your
environment).  (In our case, it's usually file.)

For the files, it's even easier.  Just have a shared fileserver that contains
all of the downloads:

PREMIRRORS_append = " \
     .*://.*/.* file://<path_to_mirror>/downloads/ \n \
"

> 
> We are planning to use Nexus and Nuget to allow storage and versioning control.
> We don’t know if these are the right tools.
> 

If you want to formalize this further, there are alternative ways to do this.
(We use an approach where we have 'download layers'.)  Between the setup program
(https://github.com/Wind-River/wr-lx-setup) and these special download layers
our users can download (to their local system) both the mirrors and tarball
downloads.

Our typical download layer looks like:

conf/layer.conf:

BBFILE_COLLECTIONS += "core-dl"
BBFILE_PATTERN_core-dl = ""
BBFILE_PATTERN_IGNORE_EMPTY_core-dl = "1"

# This should only be incremented on significant changes that will
# cause compatibility issues with other layers
LAYERVERSION_core-dl = "2.4"

# We have a pre-populated downloads directory, add to PREMIRRORS
PREMIRRORS_append = " \
     .*://.*/.* file://${LAYERDIR}/downloads/ \n \
     git://.*/.* git://${LAYERDIR}/git/MIRRORNAME;protocol=file \n \
"

Then insides of the 'downloads' directory we have a copy of all of the tarballs
used by the 'meta' layer (oe-core).  [note this can grow fairly large, so we
generally have a 2.4, then a 2.5, etc versions to keep the size managable.]

The setup program uses a local layer index to control everything.  We have a
single local patch to our oe-core that adds a 'recommendation' on this new
download layer.

LAYERRECOMMENDS_core = "core-dl (= 2.4)"


The layer index will bring this is as an 'optional' dependency which the setup
program can then use.   So if you do this for all of the layers you use -- then
a user calling setup (with our private layer index) will automatically get the
downloads that match only the layers they choose to bring into their project.

(The setup program also has a --mirror option, will will allow a local user to
mirror down the layer index contents, as well as the layers that they are using.
 This is intended both for our commercial product delivery, but also for
customers to be able to snapshot a given installation so they have all of the
code necessary to reproduce the environment in the future.)

> 
> Since we are about to embark in this project, before starting I wanted to know
> if you have some pointers on how can be implemented a mirroring framework for yocto.
> 

You can start simple, and just setup a local fileserver and git server with the
premirror setup.

You can extend this to include custom delivery and installation to customers if
this is appropriate for your environment.

(Note, the setup program is current external to OE / YP.  We are working to
integrate this into bitbake.  While there is no specific timeline promised, we
started this work in the 2.4 timeframe, and I expect much of what I explained
above will be implemented in the 2.5 timeframe.  But until then, you can use the
wr-lx-setup repository I mentioned.)

--Mark

> 
> Not sure if there are some common tools (other tools) with which yocto
> integrates nicely.
> 
>  
> 
> Any pointers will be appreciated,
> 
>  
> 
> --Hernan
> 
>  
> 
> PS> We are in morty branch currently.
> 
> 
>