[Automated-testing] Test Stack Survey
Manuel Traut
manut at linutronix.de
Fri Sep 28 08:11:45 PDT 2018
Hi Tim,
thanks for creating the survey!
On Wed, Sep 19, 2018 at 01:06:44AM +0000, Tim.Bird at sony.com wrote:
> Here are the different test systems that have representatives who have
> accepted an invitation to the summit:
> KernelCI, Fuego, Lava, SLAV, LKFT, labgrid, kerneltests, zero-day, kselftest,
> Phoronix-test-suites, LTP, r4d, ktest, Gentoo testing, Yocto testing,
> TI testing, Xilinx testing, Mentor testing, opentest.
>
> If you have a test system that is NOT on that list, and would like your
> test system to be included in our discussion or review, please complete
> this survey.
r4d is no test-system. It's just a tool for managing test-racks and control
them by libvirt.
I filled out the survey for 'ci-rt' the testsystem used by RT_PREEMPT
developers to test different kernels with different configs on various
hardwares. The ci-rt testsystem uses r4d to control the hardware.
> If you have an element that is not featured in this diagram, please let us know.
Dynamicly creating test jobs based on a test-description.
> == Survey Questions ==
> * What is the name of your test framework?
ci-rt (ci-rt.linutronix.de)
> Does your test framework:
> ==== source code access ====
> * access source code repositories for the software under test?
yes, via git
> * access source code repositories for the test software?
yes, via git
> * include the source for the test software?
test-scripts can be part of the test-description, binaries (e.g. cyclictest)
needs to be installed on the DUT.
> * provide interfaces for developers to perform code reviews?
no
> * detect that the software under test has a new version?
> ** if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.)
yes, git hook.
> * detect that the test software has a new version?
the test-description includes a git-ref of the test-software to be used.
> ==== test definitions ====
> Does your test system:
> * have a test definition repository?
> ** if so, what data format or language is used (e.g. yaml, json, shell script)
Yes, it's a folder/file layout, the files are either lists that refer to other
files, key-value pairs, kernel-patches, or kconfig overlays and kernel configs.
> Does your test definition include:
> * source code (or source code location)?
No
> * dependency information?
No
> * execution instructions?
Yes, you can specify if a kernel variant should be deployed to a target and
run e.g. cyclictest with different loads; or if it should be just bootet on
a target, or just compiled.
> * command line variants?
Yes, for kernel params and cyclictest.
> * environment variants?
Yes, for kernel compile.
> * setup instructions?
No.
> * cleanup instructions?
No.
> ** if anything else, please describe:
ci-rt is able to build and test different variants of kernels based on
different kernel configs, overlays and architectures.
> Does your test system:
> * provide a set of existing tests?
> ** if so, how many?
kernel compile, kernel boot, cyclictest, generictest
> ==== build management ====
> Does your test system:
> * build the software under test (e.g. the kernel)?
Yes and analyze compiler warnings/erros.
> * build the test software?
No.
> * build other software (such as the distro, libraries, firmware)?
No.
> * support cross-compilation?
Yes.
> * require a toolchain or build system for the SUT?
Yes, toolchains.
> * require a toolchain or build system for the test software?
N/A.
> * come with pre-built toolchains?
No. We recommend to use the toolchains that are in Debian/stretch.
> * store the build artifacts for generated software?
ci-rt generates a compile script per kernel variant. This is stored.
> ** in what format is the build metadata stored (e.g. json)?
Debian packages.
> ** are the build artifacts stored as raw files or in a database?
raw-files. Additionaly artifacts that are needed to reproduce the tests without
ci-rt are stored in a DB that is accesible by a web-application.
> *** if a database, what database?
postgresql with row-based authentication.
> ==== Test scheduling/management ====
> Does your test system:
> * check that dependencies are met before a test is run?
There is a sanity check for the test-description. Also the requirements on the
compile boxes (cross-toolchains) and on the target (cyclictest, ..) are checked.
> * schedule the test for the DUT?
Yes.
> ** select an appropriate individual DUT based on SUT or test attributes?
Yes.
> ** reserve the DUT?
Yes.
> ** release the DUT?
Yes.
> * install the software under test to the DUT?
Yes.
> * install required packages before a test is run?
No.
> * require particular bootloader on the DUT? (e.g. grub, uboot, etc.)
No.
> * deploy the test program to the DUT?
Yes. (some scripts)
> * prepare the test environment on the DUT?
Yes.
> * start a monitor (another process to collect data) on the DUT?
No.
> * start a monitor on external equipment?
No.
> * initiate the test on the DUT?
Yes.
> * clean up the test environment on the DUT?
Yes.
> ==== DUT control ====
> Does your test system:
> * store board configuration data?
> ** in what format?
R4D
> * store external equipment configuration data?
> ** in what format?
No.
> * power cycle the DUT?
Yes.
> * monitor the power usage during a run?
No.
> * gather a kernel trace during a run?
Yes.
> * claim other hardware resources or machines (other than the DUT) for use during a test?
No.
> * reserve a board for interactive use (ie remove it from automated testing)?
Yes.
> * provide a web-based control interface for the lab?
Yes / Jenkins.
> * provide a CLI control interface for the lab?
Yes / Jenkins.
> ==== Run artifact handling ====
> Does your test system:
> * store run artifacts
> ** in what format?
junit-xml
> * put the run meta-data in a database?
> ** if so, which database?
postgresql
> * parse the test logs for results?
kernel-compile, kernel-boot
> * convert data from test logs into a unified format?
> ** if so, what is the format?
junit-xml
> * evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)?
> * do you have a common set of result names: (e.g. pass, fail, skip, etc.)
> ** if so, what are they?
cyclictest threshold, no common name.
> * How is run data collected from the DUT?
> ** e.g. by pushing from the DUT, or pulling from a server?
by Jenkins stashes.
> * How is run data collected from external equipment?
serial bootlog is recorded via r4d/libvirt
> * Is external equipment data parsed?
Yes, serial bootlog is analyzed.
> ==== User interface ====
> Does your test system:
> * have a visualization system?
Yes, tomcat hosted web-application.
> * show build artifacts to users?
Yes.
> * show run artifacts to users?
Yes.
> * do you have a common set of result colors?
> ** if so, what are they?
green / red
> * generate reports for test runs?
Yes.
> * notify users of test results by e-mail?
Yes.
And Administrators about problems with the build-infrastructure. E.g. target
is not booting.
> * can you query (aggregate and filter) the build meta-data?
No.
> * can you query (aggregate and filter) the run meta-data?
No. Only graphical compare of cyclictest results.
> * what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml, etc.)
HTML, jsp
> * what language or data format is used for reports? (e.g. PDF, excel, etc.)
txt, junit-xml
> * does your test system have a CLI control tool?
> ** what is it called?
jenkins-cli
> ==== Languages: ====
> Examples: json, python, yaml, C, javascript, etc.
> * what is the base language of your test framework core?
groovy
> What languages or data formats is the user required to learn?
> (as opposed to those used internally)
the format of the ci-rt test-description
> ==== Can a user do the following with your test framework: ====
> * manually request that a test be executed (independent of a CI trigger)?
yes
> * see the results of recent tests?
yes
> * set the pass criteria for a test?
yes, for cyclictest
> ** set the threshold value for a benchmark test?
yes, for cyclictest
> ** set the list of testcase results to ignore?
no
> * provide a rating for a test? (e.g. give it 4 stars out of 5)
no
> * customize a test?
yes
> ** alter the command line for the test program?
yes
> ** alter the environment of the test program?
yes
> ** specify to skip a testcase?
yes
> ** set a new expected value for a test?
yes
> ** edit the test program source?
no
> * customize the notification criteria?
no
> ** customize the notification mechanism (eg. e-mail, text)
no
> * generate a custom report for a set of runs?
no
> * save the report parameters to generate the same report in the future?
no
> ==== Requirements ====
> Does your test framework:
> * require minimum software on the DUT?
yes
> * require minimum hardware on the DUT (e.g. memory)
no
> ** If so, what? (e.g. POSIX shell or some other interpreter, specific libraries, command line tools, etc.)
openjdk8
> * require agent software on the DUT? (e.g. extra software besides production software)
no
> ** If so, what agent?
> * is there optional agent software or libraries for the DUT?
no
> * require external hardware in your labs?
Serialserver, Powercontrol
> ==== APIS ====
> Does your test framework:
> * use existing APIs or data formats to interact within itself, or with 3rd-party modules?
junit-xml
> * have a published API for any of its sub-module interactions (any of the lines in the diagram)?
> ** Please provide a link or links to the APIs?
DUT Control:
R4D - https://github.com/ci-rt/r4d
libvirt - https://github.com/ci-rt/libvirt-debian
> Sorry - this is kind of open-ended...
> * What is the nature of the APIs you currently use?
> Are they:
> ** RPCs?
SOAP
> ** Unix-style? (command line invocation, while grabbing sub-tool output)
> ** compiled libraries?
> ** interpreter modules or libraries?
> ** web-based APIs?
> ** something else?
>
> ==== Relationship to other software: ====
> * what major components does your test framework use (e.g. Jenkins, Mondo DB, Squad, Lava, etc.)
Jenkins, R4D, libvirt, tomcat, postgresql
> * does your test framework interoperate with other test frameworks or software?
> ** which ones?
no
> == Overview ==
> Please list the major components of your test system.
>
> {{{Just as an example, Fuego can probably be divided into 3 main parts, with somewhat overlapping roles:
> * Jenkins - job triggers, test scheduling, visualization, notification
> * Core - test management (test build, test deploy, test execution, log retrieval)
> * Parser - log conversion to unified format, artifact storage, results analysis
> There are lots details omitted, but you get the idea.
> }}}
* Jenkins - job trigger, visualization, notification
ci-rt-lib (jenkins, groovy lib) - variant management, test scheduling, kernel build, kernel deploy, log/result retrieval
* libvirt/r4d - access serial console of DUT and power control DUTs
* webapplication - visualization, provide test results public (without having jenkins accesible)
* pyjutest - run commands and produce junit-xml including stdout/err return value,
runtime
More information about the automated-testing
mailing list