[yocto] gitlab-ci helper scripts for OpenEmbedded builds

[Thomas Goodwin]

The CROPS changes are all upstream now, so I've put up an example gist
here: https://gist.github.com/btgoodwin/028c3a63315a27cae3c5989455058eef.

There are comments throughout the files, and I'm still hammering out the
"deploy" stage.  There are ongoing quirks like in the case of deploying the
SDK, despite running the populate_sdk task at the start of the stage, the
related sdk directory doesn't exist or is empty as if it was not unpacked
from the shared state cache (I could not replicate this behavior at my
desk).  Another odd behavior was getting a bitbake worker halt error as if
the container ran out of space or memory, but the server doesn't agree with
that information.

One improvement I would suggest for anyone going this route would be to
have 2 separate definitions for artifacts.  The files only get touched in
the setup stage, so that should be the long-running artifact to collect
whereas all others could be trimmed back to perhaps a few minutes of keep
time.  This would prevent a copy of those artifacts being present at every
job and retained for a week (which would be a good GB for a whole pipeline
depending on how many layers and the size of the layer history).

Cheers,

Thomas


On Wed, Sep 25, 2019 at 5:17 AM Yann Dirson <yann.dirson at blade-group.com>
wrote:

> Hi Thomas,
>
> Le mar. 24 sept. 2019 à 14:36, Thomas Goodwin <btgoodwin at geontech.com> a
> écrit :
>
>> Hi Yann,
>>
>> Thanks for sharing! We're working through something similar using a tweak
>> to the CROPS docker containers and GitLab-CI (we started with autobuilder
>> 2, so we've actually merged quite a bit of that experience with our GitLab
>> setup).
>>
>
> Thanks for pointing me to CROPS, that will help bringing Docker support
> here :)
>
>
>> Using docker runners as our cluster, we setup volume mounts to the host
>> system for caching the shared state and downloads directories (as well as
>> other artifacts upon successful build).  As part of the CI setup, we then
>> insert a build/conf/auto.conf file that enforces the usage of that volume
>> (DL_DIR and SSTATE_DIR).  At the deploy stage (package feeds, etc. like
>> you're doing), we did the same thing -- copy out to that path so that it
>> ends up on the network share rather than the container.
>>
>> Also like you, we're using the "include" and "extends" instructions
>> throughout so that we have a top-level "common" set of CI YAML includes
>> alongside a growing set of repositories for each machine-specific build.
>> Each of those repositories follows the Yocto release branching style so
>> that we can trace build success to newer releases by simply creating a
>> branch and using the branch name for the initial clone of the layers.
>>
>> One thing that came directly from our autobuilder 2 experience is a way
>> to inject extra variables into the autoconf.  AB2 called this *EXTRAARGS*,
>> which was simply a list of variables to add.  For our common CI "setup"
>> stage, we check for a file of the same name (extraargs.conf) and append it
>> to the auto.conf if it exists.  In this way our downstream projects can
>> simply include that file if necessary and version control it as new release
>> branches are added.
>>
>> One of the drawbacks we've seen in going this route is that every
>> repository has its own build timeout limit, which is always laughably
>> small.  There's a backlog issue on GitLab for a global definition of this
>> value, but it's slated for 12.7 (I think).
>>
>> I'm hoping to put a write-up out soon with examples for how this all
>> worked together, but I want to hold off until we can get the CROPS changes
>> we needed upstreamed in a way that doesn't break those containers for
>> everyone else (something about the entrypoint doesn't appreciate the
>> runner's bash/shell detection script).
>>
>
> Looking forward to this!
>
>
>> Cheers,
>>
>> Thomas
>>
>> Geon Technologies, LLC
>>
>>
>> On Mon, Sep 23, 2019 at 5:50 PM Yann Dirson <yann.dirson at blade-group.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> We released our scripts to help in setting up continuous integration
>>> of a yocto-based project
>>> using Gitlab-CI.  You'll find the repository at
>>> https://github.com/BladeGroup/gitlab-oe
>>>
>>> Although we're using this in production, it still has a couple of
>>> rough edges and may need
>>> tuning for different usages.  We'd love to hear how well it fits (or
>>> doesn't fit ;) with other use
>>> cases, and will gladly consider evolutions to make it more generally
>>> usable.
>>>
>>> Best regards,
>>> --
>>> Yann Dirson <yann at blade-group.com>
>>> Blade / Shadow -- http://shadow.tech
>>> --
>>> _______________________________________________
>>> yocto mailing list
>>> yocto at yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/yocto
>>>
>>
>
> --
> Yann Dirson <yann at blade-group.com>
> Blade / Shadow -- http://shadow.tech
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20191007/98593054/attachment-0001.html>