[Automated-testing] Automatic test result interpretation

[Dan Rue]

On Thu, Jul 18, 2019 at 03:05:51PM +0200, Richard Palethorpe wrote:
> Hello,
> 
> This mainly relates to tagging failing tests with bug references (say
> Bugzilla ID or a mailing list item) and propagating those throughout the
> community. Regardless of the bug trackers, test suite, test runners or
> test result database being used.
> 
> Also automatically detecting anomalies in test results regardless of the
> test runner used and with or without nicely formatted meta data. As well
> as other similar problems.
> 
> I have, roughly speaking, created a data-analyses framework which can
> help to solve these problems and used it to create a few reports/scripts
> which handle "bug tag propagation" and some form of anomaly detection.
> 
> It is kind of difficult to explain this in writing, so please see the
> following video:
> 
> https://youtu.be/Nzha4itchg8
> https://palethorpe.gitlab.io/jdp/reports/ (link to the reports mentioned)
> 
> I designed/evolved the framework to be able to accept whatever input is
> available from many disparate sources, side stepping the issue of what
> test result format or test meta data format to use. Although it still
> requires effort to integrate a new format or accept results from a new
> test runner.
> 
> There is more information available here:
> https://palethorpe.gitlab.io/jdp/
> 
> I must stress that the methods we are using right now are quite simple,
> but we could easily incorporate something like MLJ[1]. I hypothesize
> that this would allow us to us to automate the vast majority of test
> result review. As most test failures contain some common pattern which
> can be used to identify them as a known failure. This might even be
> achievable purely by using some DSL[2] to specify heuristics.
> 
> However we first need enough data from enough sources make such efforts
> worthwhile. Currently it mostly works for our internal uses in the SUSE
> kernel QA team, but we have not had much serious interest from outside
> our team, so I will put this to one side and work one something else for
> a while to avoid tunnel vision. If you are interested, please let me
> know, so that I can justify the time to make it more accessible.
> 
> [1] https://julialang.org/blog/2019/05/beyond-ml-pipelines-with-mlj
> [2] https://palethorpe.gitlab.io/jdp/bug-tagging/#Advanced-Tagging-(Not-implemented-yet)-1

Hi Richard - thank you so much for this. Please don't treat my delay in
response as a lack of interest, I'm just getting to this corner of my
inbox now finally. I have some questions, and my notes are below which
may be useful for others.

What does JDP stand for?

You mentioned a distributed data cache and a local cache. Tell me more
:) What is it and how does it work? Sounds like this is redis but won't
it be expensive to recreate it from scratch? (Does openqa have a bulk
export ability? I know for SQUAD (LKFT's result tracker), we have to run
millions of REST queries to get everything).

Your usage of jupyter notebooks is interesting. I've played with them and
wondered if they were worth investing our time in. What problems do they solve
for you and how have they helped your team?

Why Julia? What's your and your colleagues' experience with it? What
does the learning curve look like? We're mostly a python team and I
hesitate to introduce new languages without enough advantage to cover
the overhead. I guess the provocative way to ask is, why should a python
shop consider julia?

"Edit on Github" links don't work on https://palethorpe.gitlab.io/jdp/

I watched the video on youtube (thank you!!). I'm curious JDP fits into
the overall workflow in SUSE. For example, how many users are using JDP
and the jupyter notebooks directly, vs receiving reports from it? How
does reporting automation work? I saw gitlab ci mentioned. Just
wondering what sort of traction you're seeing and what sorts of
friction.

A lot of these concepts are new to me and I'm trying to figure out how
they could fit into the daily work of my team.

Thanks,
Dan

Other notes:

JDP
- project from suse
- https://github.com/richiejp/jdp
- uses julia, jupyter notebooks
  - jupyter - list of cells: code, markdown
    - can run code cells
    - result of last command of cell will be displayed
- suse uses openqa for testing (suse project)
  - monolithic framework ("bit of a beast") - runs machine or vm, sends
    tests to them, aggregates results, etc.
- tracker support for openqa, bugzilla
  - fetch test results from openqa
  - could have a SQUAD tracker
  - will be able to have normalized test result objects to abstract out
    the result source (such as squad vs openqa)
    - this would let us aggregate tests across various projects

> 
> --
> Thank you,
> Richard.
> -- 
> _______________________________________________
> automated-testing mailing list
> automated-testing at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/automated-testing

-- 
Linaro - Kernel Validation