[Automated-testing] Test Stack Survey

[Manuel Traut]

Hi Tim,

thanks for creating the survey!

On Wed, Sep 19, 2018 at 01:06:44AM +0000, Tim.Bird at sony.com wrote:
> Here are the different test systems that have representatives who have
> accepted an invitation to the summit:
>   KernelCI, Fuego, Lava, SLAV, LKFT, labgrid, kerneltests, zero-day, kselftest,
>   Phoronix-test-suites, LTP, r4d, ktest, Gentoo testing, Yocto testing,
>   TI testing, Xilinx testing, Mentor testing, opentest.
> 
> If you have a test system that is NOT on that list, and would like your
> test system to be included in our discussion or review, please complete
> this survey.

r4d is no test-system. It's just a tool for managing test-racks and control
them by libvirt.

I filled out the survey for 'ci-rt' the testsystem used by RT_PREEMPT
developers to test different kernels with different configs on various
hardwares. The ci-rt testsystem uses r4d to control the hardware.

> If you have an element that is not featured in this diagram, please let us know.

Dynamicly creating test jobs based on a test-description.

> == Survey Questions ==
> * What is the name of your test framework?

ci-rt (ci-rt.linutronix.de)

> Does your test framework:
> ==== source code access ====
> * access source code repositories for the software under test?

yes, via git

> * access source code repositories for the test software?

yes, via git

> * include the source for the test software?

test-scripts can be part of the test-description, binaries (e.g. cyclictest)
needs to be installed on the DUT.

> * provide interfaces for developers to perform code reviews?

no

> * detect that the software under test has a new version?
> ** if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.)

yes, git hook.

> * detect that the test software has a new version?

the test-description includes a git-ref of the test-software to be used.

> ==== test definitions ====
> Does your test system:
> * have a test definition repository?
> ** if so, what data format or language is used (e.g. yaml, json, shell script)

Yes, it's a folder/file layout, the files are either lists that refer to other
files, key-value pairs, kernel-patches, or kconfig overlays and kernel configs.

> Does your test definition include:
> * source code (or source code location)?

No

> * dependency information?

No

> * execution instructions?

Yes, you can specify if a kernel variant should be deployed to a target and
run e.g. cyclictest with different loads; or if it should be just bootet on
a target, or just compiled.

> * command line variants?

Yes, for kernel params and cyclictest.

> * environment variants?

Yes, for kernel compile.

> * setup instructions?

No.

> * cleanup instructions?

No.

> ** if anything else, please describe:

ci-rt is able to build and test different variants of kernels based on
different kernel configs, overlays and architectures.

> Does your test system:
> * provide a set of existing tests?
> ** if so, how many?

kernel compile, kernel boot, cyclictest, generictest

> ==== build management ====
> Does your test system:
> * build the software under test (e.g. the kernel)?

Yes and analyze compiler warnings/erros.

> * build the test software?

No.

> * build other software (such as the distro, libraries, firmware)?

No.

> * support cross-compilation?

Yes.

> * require a toolchain or build system for the SUT?

Yes, toolchains.

> * require a toolchain or build system for the test software?

N/A.

> * come with pre-built toolchains?

No. We recommend to use the toolchains that are in Debian/stretch.

> * store the build artifacts for generated software?

ci-rt generates a compile script per kernel variant. This is stored.

> ** in what format is the build metadata stored (e.g. json)?

Debian packages.

> ** are the build artifacts stored as raw files or in a database?

raw-files. Additionaly artifacts that are needed to reproduce the tests without
ci-rt are stored in a DB that is accesible by a web-application.

> *** if a database, what database?

postgresql with row-based authentication.

> ==== Test scheduling/management ====
> Does your test system:
> * check that dependencies are met before a test is run?

There is a sanity check for the test-description. Also the requirements on the
compile boxes (cross-toolchains) and on the target (cyclictest, ..) are checked.

> * schedule the test for the DUT?

Yes.

> ** select an appropriate individual DUT based on SUT or test attributes?

Yes.

> ** reserve the DUT?

Yes.

> ** release the DUT?

Yes.

> * install the software under test to the DUT?

Yes.

> * install required packages before a test is run?

No.

> * require particular bootloader on the DUT? (e.g. grub, uboot, etc.)

No.

> * deploy the test program to the DUT?

Yes. (some scripts)

> * prepare the test environment on the DUT?

Yes.

> * start a monitor (another process to collect data) on the DUT?

No.

> * start a monitor on external equipment?

No.

> * initiate the test on the DUT?

Yes.

> * clean up the test environment on the DUT?

Yes.

> ==== DUT control ====
> Does your test system:
> * store board configuration data?
> ** in what format?

R4D

> * store external equipment configuration data?
> ** in what format?

No.

> * power cycle the DUT?

Yes.

> * monitor the power usage during a run?

No.

> * gather a kernel trace during a run?

Yes.

> * claim other hardware resources or machines (other than the DUT) for use during a test?

No.

> * reserve a board for interactive use (ie remove it from automated testing)?

Yes.

> * provide a web-based control interface for the lab?

Yes / Jenkins.

> * provide a CLI control interface for the lab?

Yes / Jenkins.

> ==== Run artifact handling ====
> Does your test system:
> * store run artifacts
> ** in what format?

junit-xml

> * put the run meta-data in a database?
> ** if so, which database?

postgresql

> * parse the test logs for results?

kernel-compile, kernel-boot

> * convert data from test logs into a unified format?
> ** if so, what is the format?

junit-xml

> * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)?
> * do you have a common set of result names: (e.g. pass, fail, skip, etc.)
> ** if so, what are they?

cyclictest threshold, no common name.

> * How is run data collected from the DUT?
> ** e.g. by pushing from the DUT, or pulling from a server?

by Jenkins stashes.

> * How is run data collected from external equipment?

serial bootlog is recorded via r4d/libvirt

> * Is external equipment data parsed?

Yes, serial bootlog is analyzed.

> ==== User interface ====
> Does your test system:
> * have a visualization system?

Yes, tomcat hosted web-application.

> * show build artifacts to users?

Yes.

> * show run artifacts to users?

Yes.

> * do you have a common set of result colors?
> ** if so, what are they?

green / red

> * generate reports for test runs?

Yes.

> * notify users of test results by e-mail?

Yes.

And Administrators about problems with the build-infrastructure. E.g. target
is not booting.

> * can you query (aggregate and filter) the build meta-data?

No.

> * can you query (aggregate and filter) the run meta-data?

No. Only graphical compare of cyclictest results.

> * what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml, etc.)

HTML, jsp

> * what language or data format is used for reports? (e.g. PDF, excel, etc.)

txt, junit-xml

> * does your test system have a CLI control tool?
> ** what is it called?

jenkins-cli

> ==== Languages: ====
> Examples: json, python, yaml, C, javascript, etc.
> * what is the base language of your test framework core?

groovy

> What languages or data formats is the user required to learn?
> (as opposed to those used internally)

the format of the ci-rt test-description

> ==== Can a user do the following with your test framework: ====
> * manually request that a test be executed (independent of a CI trigger)?

yes

> * see the results of recent tests?

yes

> * set the pass criteria for a test?

yes, for cyclictest

> ** set the threshold value for a benchmark test?

yes, for cyclictest

> ** set the list of testcase results to ignore?

no

> * provide a rating for a test? (e.g. give it 4 stars out of 5)

no

> * customize a test?

yes

> ** alter the command line for the test program?

yes

> ** alter the environment of the test program?

yes

> ** specify to skip a testcase?

yes

> ** set a new expected value for a test?

yes

> ** edit the test program source?

no

> * customize the notification criteria?

no

> ** customize the notification mechanism (eg. e-mail, text)

no

> * generate a custom report for a set of runs?

no

> * save the report parameters to generate the same report in the future?

no

> ==== Requirements ====
> Does your test framework:
> * require minimum software on the DUT?

yes

> * require minimum hardware on the DUT (e.g. memory)

no

> ** If so, what? (e.g. POSIX shell or some other interpreter, specific libraries, command line tools, etc.)

openjdk8

> * require agent software on the DUT? (e.g. extra software besides production software)

no

> ** If so, what agent?

> * is there optional agent software or libraries for the DUT?

no

> * require external hardware in your labs?

Serialserver, Powercontrol

> ==== APIS ====
> Does your test framework:
> * use existing APIs or data formats to interact within itself, or with 3rd-party modules?

junit-xml

> * have a published API for any of its sub-module interactions (any of the lines in the diagram)?
> ** Please provide a link or links to the APIs?

DUT Control:
R4D     - https://github.com/ci-rt/r4d
libvirt - https://github.com/ci-rt/libvirt-debian

> Sorry - this is kind of open-ended...
> * What is the nature of the APIs you currently use?
> Are they:
> ** RPCs?

SOAP

> ** Unix-style? (command line invocation, while grabbing sub-tool output)
> ** compiled libraries?
> ** interpreter modules or libraries?
> ** web-based APIs?
> ** something else?
> 
> ==== Relationship to other software: ====
> * what major components does your test framework use (e.g. Jenkins, Mondo DB, Squad, Lava, etc.)

Jenkins, R4D, libvirt, tomcat, postgresql

> * does your test framework interoperate with other test frameworks or software?
> ** which ones?

no

> == Overview ==
> Please list the major components of your test system.
> 
> {{{Just as an example, Fuego can probably be divided into 3 main parts, with somewhat overlapping roles:
> * Jenkins - job triggers, test scheduling, visualization, notification
> * Core - test management (test build, test deploy, test execution, log retrieval)
> * Parser - log conversion to unified format, artifact storage, results analysis
> There are lots details omitted, but you get the idea.
> }}}

 * Jenkins - job trigger, visualization, notification
ci-rt-lib (jenkins, groovy lib) - variant management, test scheduling, kernel build, kernel deploy, log/result retrieval
 * libvirt/r4d - access serial console of DUT and power control DUTs
 * webapplication - visualization, provide test results public (without having jenkins accesible)
 * pyjutest - run commands and produce junit-xml including stdout/err return value,
runtime