[Automated-testing] Test Stack Survey for tbot

[Heiko Schocher]

Hello Tim,

here the survey answers for tbot.	

== Survey Questions ==
* What is the name of your test framework?

'''tbot'''

Which of the aspects below of the CI loop does your test framework perform?

'''
Not an easy question, as tbot does commandline automatization. So in
principle, you can automate all commandline tasks ... tbot does not
differentiate between DUT or a lab PC. For example, compiling a linux
kernel (which is a testcase) can be run on the DUT (very good RAM test,
if you have mounted the rootfs over NFS) or the testcase "build linux"
can be executed on a build PC ...
'''

The answers can be: "yes", "no", or "provided by the user".
Where the answer is not simply yes or no, an explanation is appreciated.

For example, in Fuego, Jenkins is used for trigger detection (that is, to
detect new SUT versions), but the user must install the Jenkins module and
configure this themselves.  So Fuego supports triggers, but does not provide
them pre-configured for the user.

If the feature is provided by a named component in your system (or by an
external module), please provide the name of that module.

Does your test framework:
==== source code access ====
* access source code repositories for the software under test?

'''Possible, yes.'''

* access source code repositories for the test software?

'''Not needed, as the testcases are in tbot. But I can think of
writing a testcase, which clone more tests from an url ...'''

* include the source for the test software?

'''Yes.'''

* provide interfaces for developers to perform code reviews?

'''No.'''

* detect that the software under test has a new version?

'''No'''

** if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.)
* detect that the test software has a new version?

'''no.

tbot is a command line tool... the easiest CI is starting a cron job.'''

==== test definitions ====
Does your test system:
* have a test definition repository?

'''Yes'''

** if so, what data format or language is used (e.g. yaml, json, shell script)

'''The testcases are written in python.'''

Does your test definition include:
* source code (or source code location)?

'''[1] https://github.com/hsdenx/tbot/tree/master/src/tc'''

* dependency information?

'''no. Hmm... in tbot you can call testcases from testcases, so may
this could be interpretated as dependency information.'''

* execution instructions?

'''no.'''

* command line variants?

'''not in tbot itself, but possible in testcases'''

* environment variants?

'''not in tbot itself, but possible in testcases'''

* setup instructions?

'''not in tbot itself, but possible in testcases'''

* cleanup instructions?

'''not in tbot itself, but possible in testcases'''

** if anything else, please describe:

'''
May this fits into "execution instructions", I am unsure.

Tbot contains testcases for dumping register content from
"start addr" to "end address" (under linux with devmem2 in
U-Boot with md command or for the BDI debugger with "md"
command) into a file. For example a resulting pinmux file
for the BBB:

https://github.com/hsdenx/tbot/blob/master/src/files/bbb/am335x_pinmux.reg

This file can than be used in another testcase, which
reads the registers address from the file, reads the current
value from this address on the DUT and compare it with the value
in  the file. So, if you booting a new linux kernel or U-Boot, you can
check all register values, you are interested in, and be sure,
your new kernel has the same values as your reference.
'''

Does your test system:
* provide a set of existing tests?

'''Yes'''

** if so, how many?

'''
hs at xmglap:tbot  [master] $ find src/tc/ -type f | wc -l
302
hs at xmglap:tbot  [master] $

in mainline. There are more from customers, easy to add. Just create
new subdirectories in [1]. Name of testcase = filename, filename
must be unique.
'''

==== build management ====
Does your test system:
* build the software under test (e.g. the kernel)?

'''yes'''

* build the test software?

'''Hmm... tbot itself does not need to build the testcases. But you can
write testcases, which build for example a c file ... put it to
the target in someway, execute the binary ... and so on ...'''

* build other software (such as the distro, libraries, firmware)?

'''There are testcases, which checkout for example yocto, bitbake it,
and install for example the new sd card image on the DUT, check
if the new imagge boots, execute commandline tests on the new
booted image (incl. check if the new image is really the new just
created image through the /etc/timestamp yocto generates).'''

* support cross-compilation?

'''not task of tbot, but yes. I use tbot for weekly U-Boot mainline
tests (incl. automating my maintainers work). See an old video
for this tasks on youtube (no sound):

https://www.youtube.com/watch?v=zfjpj3DLsx4&feature=youtu.be
'''

* require a toolchain or build system for the SUT?

'''
not required. If you have a way for building the images you
need for your DUT, than use tbot only for installing the new images
on the DUT and test the new image.
'''

* require a toolchain or build system for the test software?

'''for tbot you need python 2.7 and python module paramiko must
be installed on the device, where you start tbot. This must not
be a big workstation, a raspberry pi for example is enough.
'''

* come with pre-built toolchains?

'''no'''

* store the build artifacts for generated software?

'''possible, if you write a testcases for it.'''

** in what format is the build metadata stored (e.g. json)?

'''currently for U-Boot simple copy raw files in a subdirectory if
all testcases finsihed successfully.

Linux / Yocto not yet (at last in mainline testcases)'''

** are the build artifacts stored as raw files or in a database?

'''In principle all is possible, if you have a command for this task
on the commandline ... write a testcase for it.'''

*** if a database, what database?

==== Test scheduling/management ====
Does your test system:
* check that dependencies are met before a test is run?

'''If the testcase writer implement such a test, yes. For example,
I started to generate an automated generated U-Boot commandline
documentation with tbot, there are testcases, which check, if
a certain U-Boot command exist. If yes, test it.

But this is the task of the testcase writer ...'''

* schedule the test for the DUT?

'''no.'''

** select an appropriate individual DUT based on SUT or test attributes?

'''no. tbot get passed through commandline arguments, what DUT in which laboratory.

Then tbot tries to get the DUT there, if this is not possilbe end with failure.
'''

** reserve the DUT?

'''You should have a possibility on the commandline to reserve the DUT,
than write a testcase for it.'''

** release the DUT?

'''See above'''

* install the software under test to the DUT?

'''yes'''

* install required packages before a test is run?

'''yes, possible'''

* require particular bootloader on the DUT? (e.g. grub, uboot, etc.)

'''no, just a commandline. For example, if your debugger has a commandline
you can automate it with tbot.'''

* deploy the test program to the DUT?

'''if your testcase needs a commmandline tool, which is not already
in the DUT rootfs, yes possible. But the testcase writer is responsible
for it, to check this. For example, if you use tbot for testing the
device before delivering to a customer, you can write a testcase,
which installs all the tools you need on the DUT, than execute all your
testcases, at the end delete all your previous installed tools.'''

* prepare the test environment on the DUT?

'''see above.'''

* start a monitor (another process to collect data) on the DUT?

'''You can open as much connections you want in tbot, and execute
testcases on this connections, so yes, possible.

For example a CAN bus test:

https://www.youtube.com/watch?v=hl7gI4b9CG8&feature=youtu.be

There a connection to a PC where a PEAK can adapter is installed is
opened. tbot connects to this PC and now you can write in your testcase
commands to this connection and commands on your DUT ...'''

* start a monitor on external equipment?

'''if this monitor has a commandline, yes.'''

* initiate the test on the DUT?

'''yes'''

* clean up the test environment on the DUT?

'''yes ... Hmm.. there is no generic way in tbot for all this (yet),
but it is possible to do.'''

==== DUT control ====
Does your test system:
* store board configuration data?

'''yes'''

** in what format?

'''
in a python file, examples:

https://github.com/hsdenx/tbot/tree/master/config
'''

* store external equipment configuration data?

'''possible'''

** in what format?

'''
add it for example in
https://github.com/hsdenx/tbot/tree/master/src/files

and add the files to git ...'''

* power cycle the DUT?

'''yes'''

* monitor the power usage during a run?

'''no'''

* gather a kernel trace during a run?

'''All output on the console is logged into a log file.'''

* claim other hardware resources or machines (other than the DUT) for use during a test?

'''yes, if this is doable through a command on a commandline.'''

* reserve a board for interactive use (ie remove it from automated testing)?

'''no'''

* provide a web-based control interface for the lab?

'''no'''

* provide a CLI control interface for the lab?

'''yes, as controlling the lab are also testcases.'''


==== Run artifact handling ====
Does your test system:
* store run artifacts

'''
no, tbot itself does not do this.

But ... There are testcases, which generate with the help of gnuplot
images (for example analyse top or latency output).

All files which tbot generates are at the end in a results directory.
'''

** in what format?

'''txt, jpg, pdf, html'''

* put the run meta-data in a database?

'''Not easy to answer... tbot itself no. But ...

You can write Event-Backends (name is not perfect). tbot calls this
backends at end of execution. In a backend, you can convert the collected
information tbot collected while runnig (so called events ... not a really
good name) into whatever format you want. An event can be for example:
- start/end of testcase
- all characters received/sended on the connections which tbot opens
- Testcases can generate Events (for example, if a U-Boot version is
   detected the testcase generate an U-Boot Version Event.

For example, there is a backend for a MySQL database, which stores some
inforamtions the user is interested in into a MySQL database. Another backend
creates a statistic image with gnuplot, another a dependency graph with
dot tool, or a jenkins backend, which creates a junit.xml for jenkins ...

see:
http://www.tbot.tools/event.html'''

** if so, which database?
* parse the test logs for results?

'''I do not know, what you mean here. In tbot you start a command on
a connection, and your testcase parses the output of the command and
decides, if it is good or bad, so yes.'''

* convert data from test logs into a unified format?

'''yes, see above.'''

** if so, what is the format?

'''http://www.tbot.tools/event.html#demos

Others are (hopefully easy) possible.'''

* evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)?

'''no. A testcase comes allways back with True or False.'''

* do you have a common set of result names: (e.g. pass, fail, skip, etc.)

'''No names. True/False only'''

** if so, what are they?

* How is run data collected from the DUT?

'''
only "console" output is logged.

But again ... if you need more, write a testcase for it ...
You can start for example  tbot instances in parallel. For example:

- start a testcases over a serial console on the DUT (some timeintensiv
   benchmark).
- start a second tbot instance, which log into the DUT over ssh and
   start on this connection a testcase, which monitors the output of
   the top command and generate at the end a graph of top values you
   are interested in with gnuplot.
'''

** e.g. by pushing from the DUT, or pulling from a server?
* How is run data collected from external equipment?

'''not used yet, but possible'''

* Is external equipment data parsed?

'''
no

Seems I have for each topic the same answer in my mind ...
(write a testcase for it ...)
'''

==== User interface ====
Does your test system:
* have a visualization system?

'''no'''

* show build artifacts to users?

'''no'''

* show run artifacts to users?

'''no'''

* do you have a common set of result colors?

'''no'''

** if so, what are they?
* generate reports for test runs?

'''yes'''

* notify users of test results by e-mail?

'''
no .. heh, but I thought about sending automated EMails, if
tbot found, that a new patch in my patchwork ToDo list does
not apply to mainline U-Boot or is not checkpatch clean ...

But I think, this should be a human task ;-)
'''

* can you query (aggregate and filter) the build meta-data?

'''no'''

* can you query (aggregate and filter) the run meta-data?

'''no'''

* what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml, 
etc.)

'''
tbot itself does not do this, it is a commandline tool, but it
can fill a MySQL database with results ... or deliver results to
jenkins, buildbot.'''

* what language or data format is used for reports? (e.g. PDF, excel, etc.)

'''simple txt log file, if no backend is enabled.'''

* does your test system have a CLI control tool?

'''yes'''

** what is it called?

'''tbot :-D '''

==== Languages: ====
Examples: json, python, yaml, C, javascript, etc.
* what is the base language of your test framework core?

'''python'''

What languages or data formats is the user required to learn?
(as opposed to those used internally)

'''python and of course, as tbot do commandline automation the
user must now the commands on the commandline.'''

==== Can a user do the following with your test framework: ====
* manually request that a test be executed (independent of a CI trigger)?

'''yes'''

* see the results of recent tests?

'''while running, yes with verbose option.'''

* set the pass criteria for a test?

'''
Heh... never thought about that, but yes!

You can pass through a commandline paramter values of variables
used in testcases ... so yes, that could be done.
'''

** set the threshold value for a benchmark test?

'''see above, yes.'''

** set the list of testcase results to ignore?

'''no'''

* provide a rating for a test? (e.g. give it 4 stars out of 5)

'''no'''

* customize a test?
** alter the command line for the test program?

'''possible if the testcase uses variables'''

** alter the environment of the test program?

'''no'''

** specify to skip a testcase?

'''no'''

** set a new expected value for a test?

'''through variables passing from command line ... yes, possible.'''

** edit the test program source?

'''Hmm.. the testcase is the testprogram ...'''

* customize the notification criteria?

'''no'''

** customize the notification mechanism (eg. e-mail, text)
* generate a custom report for a set of runs?

'''no'''

* save the report parameters to generate the same report in the future?

'''no...

Hmm... the hacker in me says yes ... the events tbot collects contains
all data ... and they are stored in background into a file ... and you
can save this file and put it into tbot with the option "-e" ... so it
should generate the same info again ...'''


==== Requirements ====
Does your test framework:
* require minimum software on the DUT?

'''depends on the testcases you want to execute!

If you have no software on the DUT, you can call (only BDI yet) testcases,
which install with a debugger Software into the DUT. Or for example
on imx based devices call:

https://github.com/hsdenx/tbot/blob/master/src/tc/linux/tc_linux_imx_usb_loader_boot_uboot.py

for using imx_usb_loader for breathing live into a device ...

But as tbot does commandline automatization, you need at least some sort
of a commandline.'''

* require minimum hardware on the DUT (e.g. memory)

'''depends on the testcases you want to execute'''

** If so, what? (e.g. POSIX shell or some other interpreter, specific libraries, command line tools, 
etc.)

'''depends on the testcases you want to execute!'''

* require agent software on the DUT? (e.g. extra software besides production software)

'''no'''

** If so, what agent?
* is there optional agent software or libraries for the DUT?

'''no'''

* require external hardware in your labs?

'''Yes. You need at least a device to which the DUT is connected. This
device is called "lab PC". This can be for example a raspberry pi.

You can of course attach more than one DUT to one lab PC.

tbot connects to this lab PC over ssh (but tbot can of course also
run on this lab PC). The lab PC must have a connection to the DUT
for the console and it must be possible to power on/off the DUT
from the lab PC. Therefore tbot always open 2 connections over
ssh to the labPC (This is the reason, why tbot uses the paramiko
modul. You can open as many connections on one ssh connection with
paramiko).

Both tasks are from tbots view just calling a testcase!

So, you need to know for your DUT on the lab PCs commandline how to

power on/off the DUT:
   -> write a tbot testcase for it
   -> use this testcase for powering on/off the DUT. You can config
      in the lab config file, which testcase is needed for the DUT.

Connect to console:
  -> write a testcase for it and use this testcase for connecting
     to the DUT. Configure this also in the lab config file.

     Remark:
     This must not be necessarily a serial line
     Of course, if your bootloader needs serial line and you want to
     do bootloader tests ... you need a serial line ... but you can
     use for example ssh testcase for connecting to the DUT.

     Also a more complicated setup (hardware reasons) I have, is possible.
     There, first I have to ssh from the lab PC to another PC and then start
     there kermit ... yes, write a testcase for this scenario and use it
     as "console testcase".
'''

==== APIS ====
Does your test framework:
* use existing APIs or data formats to interact within itself, or with 3rd-party modules?

'''no'''

* have a published API for any of its sub-module interactions (any of the lines in the diagram)?

'''no'''

** Please provide a link or links to the APIs?

Sorry - this is kind of open-ended...

'''... Sorry for my long answers ... they tend to get longer and longer ...'''

* What is the nature of the APIs you currently use?
Are they:
** RPCs?

'''no'''

** Unix-style? (command line invocation, while grabbing sub-tool output)

'''yes'''

** compiled libraries?

'''no'''

** interpreter modules or libraries?

'''no'''

** web-based APIs?

'''no'''

** something else?

'''no'''

==== Relationship to other software: ====
* what major components does your test framework use (e.g. Jenkins, Mondo DB, Squad, Lava, etc.)

'''none'''

* does your test framework interoperate with other test frameworks or software?

'''tbot can start other testframeworks if they have a commandline interface.'''

** which ones?

'''
until they are commandline based, tbot simply can call them. For example,
tbot can call the pytest based framework integrated into U-Boot:

http://git.denx.de/?p=u-boot.git;a=tree;f=test;h=7294ad8a90145a61787d251dcec741ca3a239eb2;hb=9dc8d155d4e88563f572ee79aab758eb4272f3fd

Find an output of it here:

http://xeidos.ddns.net/tbot/id_915/test-log.html

This test was started from tbot,
http://xeidos.ddns.net/tests/test_db_auslesen.php#915

and the output stored through the database backend from tbot in a mySQL database...
which ends in this html page.

Also I tried to integrate:

https://wiki.yoctoproject.org/wiki/Image_tests

as it is possible to run them from cmdline ... but I failed with:

https://lists.yoctoproject.org/pipermail/yocto/2018-January/039580.html

never got an answer ... so I did not yet finished this ... seems it is
time to try again.

So, if other testframeworks can be started from commandline, it is
easy to write a tbot testcase for them. Think about you have some
testscripts ... simply write a testcase in tbot for them (may the
testcase installs them on the DUT and removes them after the test)
and start them...
'''

== Overview ==
Please list the major components of your test system.

{{{Just as an example, Fuego can probably be divided into 3 main parts, with somewhat overlapping roles:
* Jenkins - job triggers, test scheduling, visualization, notification
* Core - test management (test build, test deploy, test execution, log retrieval)
* Parser - log conversion to unified format, artifact storage, results analysis
There are lots details omitted, but you get the idea.
}}}

Please list your major components here:
*

'''
Not easy ...
* tbot itself - open connections to lab PC on which you can start testcases
* backends - convert collected information into formats user is interested in
'''

== Glossary ==
Here is a glossary of terms.  Please indicate if your system uses different terms for these concepts.
Also, please suggest any terms or concepts that are missing.

* Bisection - automatic testing of SUT variations to find the source of a problem
* Boot - to start the DUT from an off state
* Build artifact - item created during build of the software under test
* Build manager (build server) - a machine that performs builds of the software under test
* Dependency - indicates a pre-requisite that must be filled in order for a test to run (e.g. must 
have root access, must have 100 meg of memory, some program must be installed, etc.)
* Device under test (DUT) - the hardware or product being tested (consists of hardware under test 
and software under test) (also 'board', 'target')
* Deploy - put the test program or SUT on the DUT
** this one is ambiguous - some people use this to refer to SUT installation, and others to test 
installation
* Device under Test (DUT) - a product, board or device that is being tested
* DUT controller - program and hardware for controlling a DUT (reboot, provision, etc.)
* DUT scheduler - program for managing access to a DUT (take online/offline, make available for 
interactive use)
** This is not shown in the CI Loop diagram - it could be the same as the Test Scheduler
* Lab - a collection of resources for testing one or more DUTs (also 'board farm')
* Log - one of the run artifacts - output from the test program or test framework
* Log Parsing - extracting information from a log into a machine-processable format (possibly into a 
common format)
* Monitor - a program or process to watch some attribute (e.g. power) while the test is running
** This can be on or off the DUT.
* Notification - communication based on results of test (triggered by results and including results)
* Pass criteria - set of constraints indicating pass/fail conditions for a test
* Provision (verb) - arrange the DUT and the lab environment (including other external hardware) for 
a test
** This may include installing the SUT to the device under test and booting the DUT.
* Report generation - generation of run data into a formatted output
* Request (noun) - a request to execute a test
* Result - the status indicated by a test - pass/fail (or something else) for a Run
* Results query - Selection and filtering of data from runs, to find patterns
* Run (noun) - an execution instance of a test (in Jenkins, a build)
* Run artifact - item created during a run of the test program
* Serial console - the Linux console connected over a serial connection
* Software under test (SUT) - the software being tested
* Test agent - software running on the DUT that assists in test operations (e.g. test deployment, 
execution, log gathering, debugging
** One example would be 'adb', for Android-based systems)
* Test definition - meta-data and software that comprise a particular test
* Test program - a script or binary on the DUT that performs the test
* Test scheduler - program for scheduling tests (selecting a DUT for a test, reserving it, releasing it)
* Test software - source and/or binary that implements the test
* Transport (noun) - the method of communicating and transferring data between the test system and 
the DUT
* Trigger (noun) - an event that causes the CI loop to start
* Variant - arguments or data that affect the execution and output of a test (e.g. test program 
command line; Fuego calls this a 'spec')
* Visualization - allowing the viewing of test artifacts, in aggregated form (e.g. multiple runs 
plotted in a single diagram)

bye,
Heiko
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs at denx.de