[Automated-testing] Test stack survey - Krzk Samsung-SoC

[Krzysztof Kozlowski]

Hi,

Here are the answers for my setup which I am using to boot test of
Linux kernel since few years. I use Buildbot so many answers overlap.
As Buildbot is a framework, the differences are in actual
implementation of build, boot and test steps.

My public (open-source) setup tests only Exynos-based boards but for
my employer (not Samsung) I have something similar. It's extended
version for testing other HW, QEMU and different SW. Here I described
only my public/OS setup.


== Survey Questions ==
* What is the name of your test framework?
Krzk Samsung-SoC (https://krzk.eu)

Does your test framework:
==== source code access ====
* access source code repositories for the software under test?
yes
* access source code repositories for the test software?
yes
* include the source for the test software?
yes
* provide interfaces for developers to perform code reviews?
no
* detect that the software under test has a new version?
yes
** if so, how? (e.g. polling a repository, a git hook, scanning a mail
list, etc.)
polling git repositories
* detect that the test software has a new version?
no

==== test definitions ====
Does your test system:
* have a test definition repository?
yes
** if so, what data format or language is used (e.g. yaml, json, shell script)
shell script

Does your test definition include:
* source code (or source code location)?
no
* dependency information?
no
* execution instructions?
yes
* command line variants?
no
* environment variants?
no
* setup instructions?
yes
* cleanup instructions?
yes
** if anything else, please describe:

Does your test system:
* provide a set of existing tests?
yes
** if so, how many? around 21 test cases

==== build management ====
Does your test system:
* build the software under test (e.g. the kernel)?
yes
* build the test software?
no
* build other software (such as the distro, libraries, firmware)?
no
* support cross-compilation?
yes
* require a toolchain or build system for the SUT?
yes
* require a toolchain or build system for the test software?
no
* come with pre-built toolchains?
no
* store the build artifacts for generated software?
yes
** in what format is the build metadata stored (e.g. json)?
Buildbot internal
** are the build artifacts stored as raw files or in a database?
raw files
*** if a database, what database?

==== Test scheduling/management ====
Does your test system:
* check that dependencies are met before a test is run?
no
* schedule the test for the DUT?
yes
** select an appropriate individual DUT based on SUT or test attributes?
yes
** reserve the DUT?
yes
** release the DUT?
yes
* install the software under test to the DUT?
yes/no (software is installed on the tftpboot and NFS root server, not
on the DUT)
* install required packages before a test is run?
no
* require particular bootloader on the DUT? (e.g. grub, uboot, etc.)
yes
* deploy the test program to the DUT?
no (test programs are part of NFS root so they are already deployed
when target boots)
* prepare the test environment on the DUT?
no
* start a monitor (another process to collect data) on the DUT?
yes
* start a monitor on external equipment?
no
* initiate the test on the DUT?
yes
* clean up the test environment on the DUT?
yes

==== DUT control ====
Does your test system:
* store board configuration data?
no
** in what format?
* store external equipment configuration data?
no
** in what format?
* power cycle the DUT?
yes
* monitor the power usage during a run?
no
* gather a kernel trace during a run?
yes
* claim other hardware resources or machines (other than the DUT) for
use during a test?
yes
* reserve a board for interactive use (ie remove it from automated testing)?
no
* provide a web-based control interface for the lab?
limited (Buildbot web interface)
* provide a CLI control interface for the lab?
yes (Buildbot web interface, serial console, SSH, power cycle - all
with remote access with SSH)

==== Run artifact handling ====
Does your test system:
* store run artifacts
yes
** in what format?
text (console output and kernel logs)
* put the run meta-data in a database?
yes
** if so, which database?
MySQL (handled by Buildbot/sqlalchemy)
* parse the test logs for results?
yes
* convert data from test logs into a unified format?
no
** if so, what is the format?
* evaluate pass criteria for a test (e.g. ignored results, counts or
thresholds)?
yes
* do you have a common set of result names: (e.g. pass, fail, skip, etc.)
yes
** if so, what are they?
SUCCESS, FAILURE, WARNINGS

* How is run data collected from the DUT?
reading from ssh output and serial console
** e.g. by pushing from the DUT, or pulling from a server?
* How is run data collected from external equipment?
N/A
* Is external equipment data parsed?
N/A

==== User interface ====
Does your test system:
* have a visualization system?
yes (Buildbot)
* show build artifacts to users?
no
* show run artifacts to users?
yes
* do you have a common set of result colors?
yes
** if so, what are they?
green, red, orange
* generate reports for test runs?
simplified
* notify users of test results by e-mail?
yes

* can you query (aggregate and filter) the build meta-data?
no
* can you query (aggregate and filter) the run meta-data?
no

* what language or data format is used for online results
presentation? (e.g. HTML, Javascript, xml, etc.)
no
* what language or data format is used for reports? (e.g. PDF, excel, etc.)
no

* does your test system have a CLI control tool?
no
** what is it called?

==== Languages: ====
Examples: json, python, yaml, C, javascript, etc.
* what is the base language of your test framework core?
Python, pexpect, shell

What languages or data formats is the user required to learn?
(as opposed to those used internally)

==== Can a user do the following with your test framework: ====
* manually request that a test be executed (independent of a CI trigger)?
yes
* see the results of recent tests?
no
* set the pass criteria for a test?
no
** set the threshold value for a benchmark test?
no
** set the list of testcase results to ignore?
no
* provide a rating for a test? (e.g. give it 4 stars out of 5)
no
* customize a test?
no
** alter the command line for the test program?
** alter the environment of the test program?
** specify to skip a testcase?
** set a new expected value for a test?
** edit the test program source?
* customize the notification criteria?
no
** customize the notification mechanism (eg. e-mail, text)
* generate a custom report for a set of runs?
no
* save the report parameters to generate the same report in the future?
no

==== Requirements ====
Does your test framework:
* require minimum software on the DUT?
yes/no (DUT boots from NFS root therefore except bootloader, the
minimum software has to be prepared earlier as part of preparing the
environment)
* require minimum hardware on the DUT (e.g. memory)
no (although this was not tested on HW different than few boards)
** If so, what? (e.g. POSIX shell or some other interpreter, specific
libraries, command line tools, etc.)
bootloader on target, basic operating system on NFS server
* require agent software on the DUT? (e.g. extra software besides
production software)
yes/no (SSH is required)
** If so, what agent?
* is there optional agent software or libraries for the DUT?
no
* require external hardware in your labs?
yes (for power cycle)

==== APIS ====
Does your test framework:
* use existing APIs or data formats to interact within itself, or with
3rd-party modules?
no
* have a published API for any of its sub-module interactions (any of
the lines in the diagram)?
no
** Please provide a link or links to the APIs?

Sorry - this is kind of open-ended...
* What is the nature of the APIs you currently use?
no
Are they:
** RPCs?
** Unix-style? (command line invocation, while grabbing sub-tool output)
** compiled libraries?
** interpreter modules or libraries?
** web-based APIs?
** something else?

==== Relationship to other software: ====
* what major components does your test framework use (e.g. Jenkins,
Mondo DB, Squad, Lava, etc.)
Buildbot
* does your test framework interoperate with other test frameworks or software?
no
** which ones?

== Overview ==
Please list the major components of your test system.

* Buildbot master - web interface, schedules test/builds to workers,
polls git kernel repositories,
* Buildbot workers for builds - builds kernels and uploads SUT to the master,
* Buildbot worker for boots/DUT - separate worker with multiple DUTs
connected (with serial console and power cycle abilities), fetches the
SUT and boots the DUT using it, executes the tests, gathers the output
and sends to the Buildbot master.

== Additional data ==

* https://krzk.eu - website
* https://github.com/krzk/tools/tree/master/buildbot - source code of
Buildbot configuration

Best regards,
Krzysztof