[Automated-testing] A common place for CI results?

Tue May 28 01:24:44 PDT 2019

Hello,

On Mon, May 20, 2019 at 4:38 PM Veronika Kabatova <vkabatov at redhat.com>
wrote:

>
>
> ----- Original Message -----
> > From: "Tom Gall" <tom.gall at linaro.org>
> > To: kernelci at groups.io, "Dan Rue" <dan.rue at linaro.org>
> > Cc: "Tim Bird" <Tim.Bird at sony.com>, vkabatov at redhat.com,
> automated-testing at yoctoproject.org, info at kernelci.org
> > Sent: Wednesday, May 15, 2019 11:06:33 PM
> > Subject: Re: A common place for CI results?
> >
> >
> >
> > > On May 15, 2019, at 3:33 PM, Dan Rue <dan.rue at linaro.org> wrote:
> > >
> > > On Tue, May 14, 2019 at 11:01:35PM +0000, Tim.Bird at sony.com wrote:
> > >>
> > >>
> > >>> -----Original Message-----
> > >>> From: Veronika Kabatova
> > >>>
> > >>> Hi,
> > >>>
> > >>> as we know from this list, there's plenty CI systems doing some
> testing
> > >>> on
> > >>> the
> > >>> upstream kernels (and maybe some others we don't know about).
> > >>>
> > >>> It would be great if there was a single common place where all the CI
> > >>> systems
> > >>> can put their results. This would make it much easier for the kernel
> > >>> maintainers and developers to see testing status since they only
> need to
> > >>> check one place instead of having a list of sites/mailing lists where
> > >>> each CI
> > >>> posts their contributions.
> > >>>
> > >>>
> > >>> A few weeks ago, with some people we've been talking about
> kernelci.org
> > >>> being
> > >>> in a good place to act as the central upstream kernel CI piece that
> most
> > >>> maintainers already know about. So I'm wondering if it would be
> possible
> > >>> for
> > >>> kernelci to also act as an aggregator of all results? There's
> already an
> > >>> API
> > >>> for publishing a report [0] so it shouldn't be too hard to adjust it
> to
> > >>> handle and show more information. I also found the beta version for
> test
> > >>> results [1] so actually, most of the needed functionality seems to be
> > >>> already
> > >>> there. Since there will be multiple CI systems, the source and
> contact
> > >>> point
> > >>> for the contributor (so maintainers know whom to ask about results if
> > >>> needed)
> > >>> would likely be the only missing essential data point.
> > >>>
> > >>>
> > >>> The common place for results would also make it easier for new CI
> systems
> > >>> to
> > >>> get involved with upstream. There are likely other companies out
> there
> > >>> running
> > >>> some tests on kernel internally but don't publish the results
> anywhere.
> > >>> Only
> > >>> adding some API calls into their code (with the data they are
> allowed to
> > >>> publish) would make it very simple for them to start contributing.
> If we
> > >>> want
> > >>> to make them interested, the starting point needs to be trivial.
> > >>> Different
> > >>> companies have different setups and policies and they might not be
> able
> > >>> to
> > >>> fulfill arbitrary requirements so they opt to not get involved at
> all,
> > >>> which
> > >>> is a shame because their results can be useful. After the initial
> > >>> "onboarding"
> > >>> step they might be willing to contribute more and more too.
> > >>>
> > >>>
> > >>> Please let me know if the idea makes sense or if something similar is
> > >>> already
> > >>> in plans. I'd be happy to contribute to the effort because I believe
> it
> > >>> would
> > >>> make everyone's life easier and we'd all benefit from it (and maybe
> > >>> someone
> > >>> else from my team would be willing to help out too if needed).
> > >>
> > >> I never responded to this,
> > >
> > > yea, you did. ;)
> > >
> > >> but this sounds like a really good idea to me. I don't care much which
> > >> backend we aggregate to, but it would be good as a community to start
> > >> using one service to start with.  It would help to find issues with
> > >> the API, or the results schema, if multiple people started using it.
> > >>
> > >> I know that people using Fuego are sending data to their own instances
> > >> of KernelCI.  But I don't know what the issues are for sending this
> > >> data to a shared KernelCI service.
> > >>
> > >> I would be interested in hooking up my lab to send Fuego results to
> > >> KernelCI.  This would be a good exercise.  I'm not sure what the next
> > >> steps would be, but maybe we could discuss this on the next automated
> > >> testing conference call.
> > >
> > > OK here's my idea.
> > >
> > > I don't personally think kernelci (or LKFT) are set up to aggregate
> > > results currently. We have too many assumptions about where tests are
> > > coming from, how things are built, etc. In other words, dealing with
> > > noisy data is going to be non-trivial in any existing project.
> >
> > I completely agree.
> >
>
> This is a good point. I'm totally fine with having a separate independent
> place for aggregation.
>
> > > I would propose aggregating data into something like google's BigQuery.
> > > This has a few benefits:
> > > - Non-opinionated place to hold structured data
> > > - Allows many downstream use-cases
> > > - Managed hosting, and data is publicly available
> > > - Storage is sponsored by google as a part of
> > >  https://cloud.google.com/bigquery/public-data/
> > > - First 1TB of query per 'project' is free, and users pay for more
> > >  queries than that
> >
> > I very much like this idea. I do lots of android kernel testing
> > and being able to work with / compare / contribute to what
> > is essentially a pile of data in BQ would be great. As an
> > end user working with the data I’d also have lots of dash
> > board options to customize and share queries with others.
> >
> > > With storage taken care of, how do we get the data in?
> >
> > > First, we'll need some canonical data structure defined. I would
> > > approach defining the canonical structure in conjunction with the first
> > > few projects that are interested in contributing their results. Each
> > > project will have an ETL pipeline which will extract the test results
> > > from a given project (such as kernelci, lkft, etc), translate it into
> > > the canonical data structure, and load it into the google bigquery
> > > dataset at a regular interval or in real-time. The translation layer is
> > > where things like test names are handled.
> >
>
> +1, exactly how I imagined this part.
>
> > Exactly. I would hope that the various projects that are producing
> > data would be motived to plug in. After all, it makes the data
> > they are producing more useful and available to a larger group
> > of people.
> >
> > > The things this leaves me wanting are:
> > > - raw data storage. It would be nice if raw data were stored somewhere
> > >  permanent in some intermediary place so that later implementations
> > >  could happen, and for data that doesn't fit into whatever structure we
> > >  end up with.
> >
> > I agree.
>
> +1
>
> >
> > > - time, to actually try it and find the gaps. This is just an idea I've
> > >  been thinking about. Anyone with experience here that can help flesh
> > >  this out?
> >
> > I’m willing to lend a hand.
> >
>
> Thanks for starting up a specific proposal! I agree with everything that
> was
> brought up. I'll try to find time to participate in the implementation part
> too (although my experience with data storage is.. limited, I should be
> able
> to help out with the structure prototyping and maybe other parts too).
>

This all sounds great: having a common location to store the
results that is scalable and definitions of test case names.

However, there is a whole layer of logic above and around this
which KernelCI does, and I'm sure other CI systems also do with
some degree of overlap.  So it seems to me that solving how to
deal with the results is only one piece in the puzzle to get a
common CI architecture for upstream kernel testing.  Sorry I'm a
bit late to the party so I'll add my 2¢ here...

Around the end of last year I made this document and mentioned it
on this list, about making KernelCI more modular:

https://docs.google.com/document/d/15F42HdHTO6NbSL53_iLl77lfe1XQKdWaHAf7XCNkKD8/edit?usp=sharing
  https://groups.io/g/kernelci/topic/kernelci_modular_pipeline/29692355

The idea is to make it possible to have alternative components in
the KernelCI "pipeline".  Right now, KernelCI has these
components:

* Jenkins job to monitor git branches and build kernels
* LAVA to run tests
* Custom backend and storage server to keep binaries and data
* Custom web frontend to show the results

They could all be replaced or used in conjunction with
alternative build systems, database engines, test lab schedulers
and dashboards.  The key thing is the code orchestrating all
this, which is kept in the kernelci-core repository.

For example, when a change has been detected in a tree, rather
than triggering kernel builds on Jenkins there could be a request
sent to another build system to do that elsewhere.  Likewise,
when some builds are ready to be tested, jobs could be scheduled
in non-LAVA labs simply by sending another kind of HTTP request
than the ones we're currently sending to the LAVA APIs.  This
could be easily described in some config files, in fact we
already have one with the list of labs where to submit jobs to.
Builds and tests are configured in YAML files in KernelCI, which
could easily be extended too with new attributes.

The big advantage of having a central way to orchestrate all this
is that results are going to be consistent and higher-level
features can be enabled: each tree will be sampled at the same
commit, so we don't end up with one CI lab running a version of
mainline a few patches than another one etc...  It means we can
expand some KernelCI features to a larger ecosystem of CI labs,
such as:

* redundancy checks when the same test / hardware is tested in
  multiple places (say, if a RPi fails to boot in a LAVA lab but
  not in CKI's lab...)

* regression tracking across the whole spectrum of CI labs

* common reports for each single kernel revision being tested

* bisections extended to non-LAVA labs

It feels like a diagram would be needed to really give an idea of
how this would work.  APIs and callback mechanisms would need to
be well defined to have clear entry points for the various
components in a modular system like this.  I think we would be
able to reuse some of the things currently used by KernelCI and
improve them, taking into account what other CI labs have been
doing (LKFT, CKI...).

I'm only scratching the surface here, but I wanted to raise this
point to see if others shared the same vision.  It would be
unfortunate if we came up with a great solution focused on
results, but then realised that it had big design limitations
when trying to add more abstract functionality across all the
contributing CI labs.

Well, I tried to keep it short - hope this makes any sense.

Cheers,
Guillaume
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/automated-testing/attachments/20190528/377259c8/attachment.html>