[Automated-testing] CKI Test Stack Survey (was RE: CKI project introduction)

[Veronika Kabatova]

----- Original Message -----
> From: "Tim Bird" <Tim.Bird at sony.com>
> To: vkabatov at redhat.com
> Cc: automated-testing at yoctoproject.org
> Sent: Wednesday, August 7, 2019 9:10:08 PM
> Subject: RE: CKI Test Stack Survey (was RE: CKI project introduction)
> 
> Thanks!
> 
> Sorry this took so long for me to process (only 5 months!).
> 
> I've created the following wiki page for the survey response:
> https://elinux.org/CKI_survey_response
> 
> And it's linked from the main test stack survey page here:
> https://elinux.org/Test_Stack_Survey
> 

Thanks! A lot of things changed since I filled out the initial info. I think
I already have access to wiki editing so I'll just go ahead and update the
page so I don't have to give you more work :)


> It's quite interesting to see how CKI compares with other systems.
> I'm not sure if it's a one-to-one correspondence, but I get the impression
> that CKI/Beaker is comparable in division of responsibilities to LKFT/LAVA.
> 
> It would be interesting to do a comparison between Beaker and LAVA.
> They are quite different in implementation, although at a high level
> I think they do similar things.
> 

I'm not familiar with LAVA except "it runs tests" but if you want any info
about Beaker feel free to ask and I'll do my best to answer everything.

> I added an "additional information" section.  Can you take a look and
> make sure that what I have there is correct.  For example, I think your
> main project web site is at:
> https://cki-project.org
> 
> with project source repositories at:
> https://github.com/CKI-project
> 
> and I think your open source tests are at:
> https://github.com/CKI-project/tests-beaker
> 
> Can you recommend some good example tests from your collection, to examine
> to get a feel for your system?
> 
> Specifically, if you have a very simple test, and a "characteristic" test
> that you
> could point to (give the repository URL for each test), that would help me
> understand
> your test definition (which is what I'm working on now, in preparation for
> the
> meetings in Lisbon).
> 

The tests we have are maintained by kernel QE and the interface they use to
talk to Beaker varies a lot from team to team. However we do actually have
a super simple example test -- basically "exit 0" wrapped as Beaker task [0].

A test that is still very simple but uses more Beaker features (result
reporting, log uploads etc.) is [1]. A more complicated one is the LTP Lite
wrapper which includes a knownissue detection and spans into a different
include directory.

Most test definition information can be found in "Makefile" and "metadata"
(if present) files. We also have an internal test definition "DB" which we
are looking into open sourcing but didn't get that far yet. This one
contains information about test maintainers, test waiving, which trees and
architectures the test should be executed, which modified files in kernel
should trigger the test etc. Some of these information is also present in
the data/code of the tests themselves but not all.


Let me know if this helps or if you need more info. I'll try to update the
info page later today.

Veronika


[0] https://github.com/CKI-project/tests-beaker/tree/master/example
[1] https://github.com/CKI-project/tests-beaker/tree/master/acpi/acpitable
[2] https://github.com/CKI-project/tests-beaker/tree/master/distribution/ltp/lite

> Thanks,
>  -- Tim
> 

<snipped the long survey part>