[Automated-testing] A common place for CI results?

[Veronika Kabatova]

----- Original Message ----- 

> From: "Guillaume Tucker" <guillaume.tucker at gmail.com>
> To: kernelci at groups.io, vkabatov at redhat.com
> Cc: "Tom Gall" <tom.gall at linaro.org>, "Dan Rue" <dan.rue at linaro.org>, "Tim
> Bird" <Tim.Bird at sony.com>, automated-testing at yoctoproject.org,
> info at kernelci.org
> Sent: Tuesday, May 28, 2019 10:24:44 AM
> Subject: Re: A common place for CI results?

> Hello,

> On Mon, May 20, 2019 at 4:38 PM Veronika Kabatova < vkabatov at redhat.com >
> wrote:

> > ----- Original Message -----
> 
> > > From: "Tom Gall" < tom.gall at linaro.org >
> 
> > > To: kernelci at groups.io , "Dan Rue" < dan.rue at linaro.org >
> 
> > > Cc: "Tim Bird" < Tim.Bird at sony.com >, vkabatov at redhat.com ,
> > > automated-testing at yoctoproject.org , info at kernelci.org
> 
> > > Sent: Wednesday, May 15, 2019 11:06:33 PM
> 
> > > Subject: Re: A common place for CI results?
> 
> > >
> 
> > >
> 
> > >
> 
> > > > On May 15, 2019, at 3:33 PM, Dan Rue < dan.rue at linaro.org > wrote:
> 
> > > >
> 
> > > > On Tue, May 14, 2019 at 11:01:35PM +0000, Tim.Bird at sony.com wrote:
> 
> > > >>
> 
> > > >>
> 
> > > >>> -----Original Message-----
> 
> > > >>> From: Veronika Kabatova
> 
> > > >>>
> 
> > > >>> Hi,
> 
> > > >>>
> 
> > > >>> as we know from this list, there's plenty CI systems doing some
> > > >>> testing
> 
> > > >>> on
> 
> > > >>> the
> 
> > > >>> upstream kernels (and maybe some others we don't know about).
> 
> > > >>>
> 
> > > >>> It would be great if there was a single common place where all the CI
> 
> > > >>> systems
> 
> > > >>> can put their results. This would make it much easier for the kernel
> 
> > > >>> maintainers and developers to see testing status since they only need
> > > >>> to
> 
> > > >>> check one place instead of having a list of sites/mailing lists where
> 
> > > >>> each CI
> 
> > > >>> posts their contributions.
> 
> > > >>>
> 
> > > >>>
> 
> > > >>> A few weeks ago, with some people we've been talking about
> > > >>> kernelci.org
> 
> > > >>> being
> 
> > > >>> in a good place to act as the central upstream kernel CI piece that
> > > >>> most
> 
> > > >>> maintainers already know about. So I'm wondering if it would be
> > > >>> possible
> 
> > > >>> for
> 
> > > >>> kernelci to also act as an aggregator of all results? There's already
> > > >>> an
> 
> > > >>> API
> 
> > > >>> for publishing a report [0] so it shouldn't be too hard to adjust it
> > > >>> to
> 
> > > >>> handle and show more information. I also found the beta version for
> > > >>> test
> 
> > > >>> results [1] so actually, most of the needed functionality seems to be
> 
> > > >>> already
> 
> > > >>> there. Since there will be multiple CI systems, the source and
> > > >>> contact
> 
> > > >>> point
> 
> > > >>> for the contributor (so maintainers know whom to ask about results if
> 
> > > >>> needed)
> 
> > > >>> would likely be the only missing essential data point.
> 
> > > >>>
> 
> > > >>>
> 
> > > >>> The common place for results would also make it easier for new CI
> > > >>> systems
> 
> > > >>> to
> 
> > > >>> get involved with upstream. There are likely other companies out
> > > >>> there
> 
> > > >>> running
> 
> > > >>> some tests on kernel internally but don't publish the results
> > > >>> anywhere.
> 
> > > >>> Only
> 
> > > >>> adding some API calls into their code (with the data they are allowed
> > > >>> to
> 
> > > >>> publish) would make it very simple for them to start contributing. If
> > > >>> we
> 
> > > >>> want
> 
> > > >>> to make them interested, the starting point needs to be trivial.
> 
> > > >>> Different
> 
> > > >>> companies have different setups and policies and they might not be
> > > >>> able
> 
> > > >>> to
> 
> > > >>> fulfill arbitrary requirements so they opt to not get involved at
> > > >>> all,
> 
> > > >>> which
> 
> > > >>> is a shame because their results can be useful. After the initial
> 
> > > >>> "onboarding"
> 
> > > >>> step they might be willing to contribute more and more too.
> 
> > > >>>
> 
> > > >>>
> 
> > > >>> Please let me know if the idea makes sense or if something similar is
> 
> > > >>> already
> 
> > > >>> in plans. I'd be happy to contribute to the effort because I believe
> > > >>> it
> 
> > > >>> would
> 
> > > >>> make everyone's life easier and we'd all benefit from it (and maybe
> 
> > > >>> someone
> 
> > > >>> else from my team would be willing to help out too if needed).
> 
> > > >>
> 
> > > >> I never responded to this,
> 
> > > >
> 
> > > > yea, you did. ;)
> 
> > > >
> 
> > > >> but this sounds like a really good idea to me. I don't care much which
> 
> > > >> backend we aggregate to, but it would be good as a community to start
> 
> > > >> using one service to start with. It would help to find issues with
> 
> > > >> the API, or the results schema, if multiple people started using it.
> 
> > > >>
> 
> > > >> I know that people using Fuego are sending data to their own instances
> 
> > > >> of KernelCI. But I don't know what the issues are for sending this
> 
> > > >> data to a shared KernelCI service.
> 
> > > >>
> 
> > > >> I would be interested in hooking up my lab to send Fuego results to
> 
> > > >> KernelCI. This would be a good exercise. I'm not sure what the next
> 
> > > >> steps would be, but maybe we could discuss this on the next automated
> 
> > > >> testing conference call.
> 
> > > >
> 
> > > > OK here's my idea.
> 
> > > >
> 
> > > > I don't personally think kernelci (or LKFT) are set up to aggregate
> 
> > > > results currently. We have too many assumptions about where tests are
> 
> > > > coming from, how things are built, etc. In other words, dealing with
> 
> > > > noisy data is going to be non-trivial in any existing project.
> 
> > >
> 
> > > I completely agree.
> 
> > >
> 

> > This is a good point. I'm totally fine with having a separate independent
> 
> > place for aggregation.
> 

> > > > I would propose aggregating data into something like google's BigQuery.
> 
> > > > This has a few benefits:
> 
> > > > - Non-opinionated place to hold structured data
> 
> > > > - Allows many downstream use-cases
> 
> > > > - Managed hosting, and data is publicly available
> 
> > > > - Storage is sponsored by google as a part of
> 
> > > > https://cloud.google.com/bigquery/public-data/
> 
> > > > - First 1TB of query per 'project' is free, and users pay for more
> 
> > > > queries than that
> 
> > >
> 
> > > I very much like this idea. I do lots of android kernel testing
> 
> > > and being able to work with / compare / contribute to what
> 
> > > is essentially a pile of data in BQ would be great. As an
> 
> > > end user working with the data I’d also have lots of dash
> 
> > > board options to customize and share queries with others.
> 
> > >
> 
> > > > With storage taken care of, how do we get the data in?
> 
> > >
> 
> > > > First, we'll need some canonical data structure defined. I would
> 
> > > > approach defining the canonical structure in conjunction with the first
> 
> > > > few projects that are interested in contributing their results. Each
> 
> > > > project will have an ETL pipeline which will extract the test results
> 
> > > > from a given project (such as kernelci, lkft, etc), translate it into
> 
> > > > the canonical data structure, and load it into the google bigquery
> 
> > > > dataset at a regular interval or in real-time. The translation layer is
> 
> > > > where things like test names are handled.
> 
> > >
> 

> > +1, exactly how I imagined this part.
> 

> > > Exactly. I would hope that the various projects that are producing
> 
> > > data would be motived to plug in. After all, it makes the data
> 
> > > they are producing more useful and available to a larger group
> 
> > > of people.
> 
> > >
> 
> > > > The things this leaves me wanting are:
> 
> > > > - raw data storage. It would be nice if raw data were stored somewhere
> 
> > > > permanent in some intermediary place so that later implementations
> 
> > > > could happen, and for data that doesn't fit into whatever structure we
> 
> > > > end up with.
> 
> > >
> 
> > > I agree.
> 

> > +1
> 

> > >
> 
> > > > - time, to actually try it and find the gaps. This is just an idea I've
> 
> > > > been thinking about. Anyone with experience here that can help flesh
> 
> > > > this out?
> 
> > >
> 
> > > I’m willing to lend a hand.
> 
> > >
> 

> > Thanks for starting up a specific proposal! I agree with everything that
> > was
> 
> > brought up. I'll try to find time to participate in the implementation part
> 
> > too (although my experience with data storage is.. limited, I should be
> > able
> 
> > to help out with the structure prototyping and maybe other parts too).
> 

> This all sounds great: having a common location to store the
> results that is scalable and definitions of test case names.

> However, there is a whole layer of logic above and around this
> which KernelCI does, and I'm sure other CI systems also do with
> some degree of overlap. So it seems to me that solving how to
> deal with the results is only one piece in the puzzle to get a
> common CI architecture for upstream kernel testing. Sorry I'm a
> bit late to the party so I'll add my 2¢ here...

> Around the end of last year I made this document and mentioned it
> on this list, about making KernelCI more modular:

> https://docs.google.com/document/d/15F42HdHTO6NbSL53_iLl77lfe1XQKdWaHAf7XCNkKD8/edit?usp=sharing
> https://groups.io/g/kernelci/topic/kernelci_modular_pipeline/29692355

I have a plan to read this document but still didn't get around it :(


> The idea is to make it possible to have alternative components in
> the KernelCI "pipeline". Right now, KernelCI has these
> components:

> * Jenkins job to monitor git branches and build kernels
> * LAVA to run tests
> * Custom backend and storage server to keep binaries and data
> * Custom web frontend to show the results

> They could all be replaced or used in conjunction with
> alternative build systems, database engines, test lab schedulers
> and dashboards. The key thing is the code orchestrating all
> this, which is kept in the kernelci-core repository.

> For example, when a change has been detected in a tree, rather
> than triggering kernel builds on Jenkins there could be a request
> sent to another build system to do that elsewhere. Likewise,
> when some builds are ready to be tested, jobs could be scheduled
> in non-LAVA labs simply by sending another kind of HTTP request
> than the ones we're currently sending to the LAVA APIs. This
> could be easily described in some config files, in fact we
> already have one with the list of labs where to submit jobs to.
> Builds and tests are configured in YAML files in KernelCI, which
> could easily be extended too with new attributes.

> The big advantage of having a central way to orchestrate all this
> is that results are going to be consistent and higher-level
> features can be enabled: each tree will be sampled at the same
> commit, so we don't end up with one CI lab running a version of
> mainline a few patches than another one etc... It means we can
> expand some KernelCI features to a larger ecosystem of CI labs,
> such as:

> * redundancy checks when the same test / hardware is tested in
> multiple places (say, if a RPi fails to boot in a LAVA lab but
> not in CKI's lab...)

> * regression tracking across the whole spectrum of CI labs

> * common reports for each single kernel revision being tested

> * bisections extended to non-LAVA labs

> It feels like a diagram would be needed to really give an idea of
> how this would work. APIs and callback mechanisms would need to
> be well defined to have clear entry points for the various
> components in a modular system like this. I think we would be
> able to reuse some of the things currently used by KernelCI and
> improve them, taking into account what other CI labs have been
> doing (LKFT, CKI...).

> I'm only scratching the surface here, but I wanted to raise this
> point to see if others shared the same vision. It would be
> unfortunate if we came up with a great solution focused on
> results, but then realised that it had big design limitations
> when trying to add more abstract functionality across all the
> contributing CI labs.

This is definitely something I'd love to see in the (likely very far)
future too.

However, I'd say that it would require changes in the CI systems and not
necessarily in the result format / way of displaying the data, which is
what we are trying to set up and agree on here. Each of the CI systems in
question would be responsible for their API calls and the receiving side
would just validate what it got, and this is likely not something that
would change even in the far future when we get more integrated.


I agree with all the points you made and we should definitely keep them
in mind going forward, but unless I overlooked something (which is
totally possible :) these two things don't depend on each other.

> Well, I tried to keep it short - hope this makes any sense.

It does. I really have to find some time to get through that doc.


Veronika

> Cheers,
> Guillaume