[Automated-testing] Gentoo Kernel CI survey response

[Alice Ferrazzi]

= Gentoo Kernel CI survey response =
Gentoo Kernel CI survey response provided by Alice Ferrazzi

Note that Gentoo Kernel CI is heavily based on Buildbot.

== Survey Questions ==
* What is the name of your test framework?
[https://github.com/gentoo/Gentoo_kernelCI Gentoo Kernel CI]

Does your test framework:
==== source code access ====
* access source code repositories for the software under test? '''yes'''
* access source code repositories for the test software? '''yes'''
* include the source for the test software? '''yes'''
* provide interfaces for developers to perform code reviews? '''yes'''
* detect that the software under test has a new version? '''yes'''
** if so, how? '''git polling'''
* detect that the test software has a new version? '''experimental'''

==== test definitions ====
Does your test system:
* have a test definition repository? '''yes'''
** if so, what data format or language is used? '''Python/Bash'''

Does your test definition include:
* source code (or source code location)? yes
 [https://github.com/gentoo/Gentoo_kernelCI
https://github.com/gentoo/Gentoo_kernelCI]
* dependency information? '''Are part of buildbot, not yet defined for
Gentoo Kernel CI'''
* execution instructions? yes
* command line variants? yes
* environment variants? yes
* setup instructions? yes
* cleanup instructions? yes
** if anything else, please describe:

'''Gentoo Kernel CI uses [http://buildbot.net/ Buildbot] for test
definitions and running tests, so please refer to
[http://buildbot.net/ Buildbot] for responses.'''


Does your test system:
* provide a set of existing tests?
** if so, how many?  '''Gentoo Kernel CI is checking that the Kernel
boots correctly with differents kernel configuration.'''

==== build management ====
Does your test system:

* build the software under test (e.g. the kernel)? '''yes'''
* build the test software? '''yes'''
* build other software (such as the distro, libraries, firmware)? '''yes'''
* support cross-compilation? '''yes''
* require a toolchain or build system for the SUT? '''yes'''
* require a toolchain or build system for the test software? '''no'''
* come with pre-built toolchains? '''yes (part of Gentoo)'''
* store the build artifacts for generated software?
** in what format is the build metadata stored (e.g. json)?
** are the build artifacts stored as raw files or in a database?
*** if a database, what database? everything supported by sqlalchemy

==== Test scheduling/management ====
Does your test system:
* check that dependencies are met before a test is run?
* schedule the test for the DUT? '''yes'''
** select an appropriate individual DUT based on SUT or test attributes?
** reserve the DUT? '''yes'''
** release the DUT? '''yes'''
* install the software under test to the DUT?
* install required packages before a test is run? '''yes/only kernel packages'''
* require particular bootloader on the DUT? (e.g. grub, uboot, etc.)
'''Working with qemu as now, real hardware is still in progress'''
* deploy the test program to the DUT?
* prepare the test environment on the DUT?
* start a monitor (another process to collect data) on the DUT?
* start a monitor on external equipment?
* initiate the test on the DUT? '''yes'''
* clean up the test environment on the DUT? '''yes'''

==== DUT control ====
Does your test system:
* store board configuration data?
** in what format?
* store external equipment configuration data?
** in what format?
* power cycle the DUT?
* monitor the power usage during a run?
* gather a kernel trace during a run?
* claim other hardware resources or machines (other than the DUT) for
use during a test?
* reserve a board for interactive use (ie remove it from automated testing)?
* provide a web-based control interface for the lab? '''Buildbot web
interface'''
* provide a CLI control interface for the lab? '''yes, Buildbot-worker'''

'''Gentoo kernel CI is working with Qemu container, real hardware is
still in progress.'''

==== Run artifact handling ====
Does your test system:
* store run artifacts '''yes'''
** in what format? '''raw files'''
* put the run meta-data in a database? '''yes'''
** if so, which database? '''anything supported by sqlalchemy'''
* parse the test logs for results? '''yes'''
* convert data from test logs into a unified format? '''no'''
** if so, what is the format?
* evaluate pass criteria for a test (e.g. ignored results, counts or
thresholds)? '''yes'''
* do you have a common set of result names: (e.g. pass, fail, skip,
etc.) '''yes'''
** if so, what are they? '''Gentoo Kernel CI is using
[http://docs.buildbot.net/latest/developer/results.html Buildbot test
results]'''

* How is run data collected from the DUT?
'''Buildbot-worker/Buildbot master'''
** e.g. by pushing from the DUT, or pulling from a server?

* How is run data collected from external equipment?
* Is external equipment data parsed?

==== User interface ====
Does your test system:
* have a visualization system? '''yes'''
* show build artifacts to users? '''yes'''
* show run artifacts to users? '''yes'''
* do you have a common set of result colors? '''yes'''
** if so, what are they?
'''[http://docs.buildbot.net/latest/developer/results.html Buildbot
test results]'''
* generate reports for test runs? '''yes'''
* notify users of test results by e-mail? '''yes'''

* can you query (aggregate and filter) the build meta-data? '''yes'''
* can you query (aggregate and filter) the run meta-data? '''yes'''

* what language or data format is used for online results
presentation? '''custom html/javascript app'''
* what language or data format is used for reports? '''Depend from the
report tool used'''

* does your test system have a CLI control tool? '''yes'''
** what is it called? '''buildbot"

==== Languages: ====
Examples: json, python, yaml, C, javascript, etc.
* what is the base language of your test framework core?
'''Python'''
* What languages or data formats is the user required to learn? '''Python'''


==== Can a user do the following with your test framework: ====
* Can a user manually request that a test be executed? '''yes'''
* see the results of recent tests? '''yes'''
* set the pass criteria for a test? '''yes'''
** set the threshold value for a benchmark test? '''not sure'''
** set the list of testcase results to ignore? '''not sure'''
* provide a rating for a test? (e.g. give it 4 stars out of 5) '''no'''
* customize a test? '''yes'''
** alter the command line for the test program? '''no'''
** alter the environment of the test program? '''yes'''
** specify to skip a testcase? '''yes'''
** set a new expected value for a test? '''yes'''
** edit the test program source? '''yes'''
* customize the notification criteria? '''yes'''
** customize the notification mechanism (eg. e-mail, text) '''yes'''
* generate a custom report for a set of runs? '''yes'''
* save the report parameters to generate the same report in the
future? '''yes'''

==== Requirements ====
Does your test framework:
* require minimum software on the DUT?
* require minimum hardware on the DUT (e.g. memory)
** If so, what?

* require agent software on the DUT?  '''yes'''
** If so, what agent? '''buildbot-worker'''
* is there optional agent software or libraries for the DUT? '''no'''
* require external hardware in your labs? '''no'''

==== APIS ====
Does your test framework:
* use existing APIs or data formats to interact within itself, or with
3rd-party modules? '''yes
[http://docs.buildbot.net/latest/developer/apis.html]'''
* have a published API for any of its sub-module interactions (any of
the lines in the diagram)? '''yes'''
** Please provide a link or links to the APIs?
'''[http://docs.buildbot.net/latest/developer/apis.html]'''

* What is the nature of the APIs you currently use?
Are they:
** RPCs?
** Unix-style?
** compiled libraries?
** interpreter modules or libraries?
** web-based APIs?
** something else?

==== Relationship to other software: ====
* what major components does your test framework use? '''Buildbot'''
* does your test framework interoperate with other test frameworks or
software? '''yes'''
** which ones? '''Buildbot'''

== Overview ==
Please list your major components here:
* '''Build/Test Management (Buildbot)'''
* '''Test Scheduling (Buildbot)'''
* '''DUT Control (Gentoo Kernel CI)'''
* '''Results Management (Buildbot)'''
* '''View/Interact (Buildbot)'''

= Additional Data =


Please ask for any clarifications
-- 
Thanks,
Alice Ferrazzi

Gentoo Kernel Project Leader
Gentoo Foundation Vice-Secretary
Gentoo Google Summer of Code Administrator
Mail: Alice Ferrazzi <alicef at gentoo.org>
PGP: 2E4E 0856 461C 0585 1336 F496 5621 A6B2 8638 781A