[Automated-testing] Automated Testing Summit - Test Stack survey

[Guenter Roeck]

On Tue, Sep 18, 2018 at 5:07 PM <Tim.Bird at sony.com> wrote:

> Hello invitee to the Automated Testing Summit,
>
> Executive Summary:  Here's a survey. If you "own" one of the test systems
> mentioned below, please fill it out before the summit (by Oct 2).
>
> This e-mail is to notify you that plans are proceeding with the Automated
> Testing Summit,
> and Kevin and I look forward to your participation at the event.
>
> In preparation for our discussions, we have prepared a diagram of the
> Continuous Integration
> Loop, and a survey to ask you questions about your respective test systems.
>
> We'd like to conduct the survey in public, but you can respond in private
> if you'd like.
> We'd like to get just one survey response per test system.  Some test
> systems have
> more than one representative coming to the summit.  So, if you see someone
> else
> working on your test system in the To: list above, please coordinate with
> that person
> to have one of you answer the survey.
>
> I may be missing something (if so I apologize), but here are the different
> test
> systems I expect represented at the event:
>   KernelCI, Fuego, Lava, SLAV, LKFT, labgrid, kerneltests, zero-day,
> kselftest,
>   Phoronix-test-suites, LTP, r4d, ktest, Gentoo testing, Yocto testing,
>   TI testing, Xilinx testing, Mentor testing, opentest, tbot.
>
> Please send the survey response to Kevin and me, and
> CC: <automated-testing at yoctoproject.org>
> The reasons for posting to the automated-testing list is so that we can
> archive the answer and any ensuing discussion clarifying how your test
> system works.  Please note we will also be sending the survey to that
> list, for
> public responses on any test systems not represented at the summit.
>
> Just to put your mind not-at-ease about speaking at the event...
> We are still working on the exact format of the event, but I expect that
> for many of the
> test systems listed above we will want a VERY short (5 minute) lightning
> talk on
> 1) the status of your project,
> 2) the differences between our reference CI loop and how your system works,
> 3) any special features your system has that others don't.
>
> We want to leave lots of time for open discussions.
>
> Please complete the survey no later than October 2 (two weeks from now).
>
> Sorry for being late. Feel free to drop/ignore.


> OK - finally, here is the survey.  Sorry it's so long, but many questions
> should be easy to answer.
> Please refer to the attached diagram, or see the page:
> https://elinux.org/Test_Stack_Survey.
>
> The format of the survey is mediawiki markup, so * = bullet, ** = indented
> bullet.
>
> == Diagrams ==
> Attached is a diagram for the high level CI loop:
>
> The boxes represent different processes, hardware, or storage locations.
> Lines between boxes indicate APIs or control flow,
> and are labeled with letters.  The intent of this is to facilitate
> discussion at the summit.
>
> [[File:<see attachment>|high level CI loop]]
> --------
>
> If you have an element that is not featured in this diagram, please let us
> know.
>
> == Cover text ==
> Hello Test Framework developer or user,
>
> The purpose of this survey is to try to understand how different Test
> Frameworks and Automated Test components
> in the Linux Test ecosystem work - what features they have, what
> terminology they use, and so forth.
> The reason to characterize these different pieces of software (and
> hardware) is to try to come up
> with definitions for a Test Stack, and possibly API definitions, that will
> allow different
> elements to communicate and interact.  We are interested in seeing the
> commonalities and differences
> between stack elements.
>
> This information will be used, to start, to prepare for discussions about
> test stack standards
> at the Automated Testing Summit 2018.
>
> Please see the Glossary below for the meaning of words used in this
> survey.  If you use different
> words in your framework for the same concept, please let us know.  If you
> think there are other
> words that should be in the Glossary, please let us know.
>
> == Survey Questions ==
> * What is the name of your test framework?
>
> kerneltests.org


> Which of the aspects below of the CI loop does your test framework perform?
>
> The answers can be: "yes", "no", or "provided by the user".
> Where the answer is not simply yes or no, an explanation is appreciated.
>
> For example, in Fuego, Jenkins is used for trigger detection (that is, to
> detect new SUT versions), but the user must install the Jenkins module and
> configure this themselves.  So Fuego supports triggers, but does not
> provide
> them pre-configured for the user.
>
> If the feature is provided by a named component in your system (or by an
> external module), please provide the name of that module.
>
> Does your test framework:
> ==== source code access ====
> * access source code repositories for the software under test?
>
yes


> * access source code repositories for the test software?
>
no


> * include the source for the test software?
>
yes


> * provide interfaces for developers to perform code reviews?
>

no.


> * detect that the software under test has a new version?
>

yes


> ** if so, how? (e.g. polling a repository, a git hook, scanning a mail
> list, etc.)
>

git hooks


> * detect that the test software has a new version?
>
> No.


> ==== test definitions ====
> Does your test system:
> * have a test definition repository?
> ** if so, what data format or language is used (e.g. yaml, json, shell
> script)
>
> No


> Does your test definition include:
> * source code (or source code location)?
>

https://github.com/groeck/linux-build-test


> * dependency information?
>

Not sure I understand what that is.


> * execution instructions?
>

yes


> * command line variants?
>

no


> * environment variants?
>

not under user control.


> * setup instructions?
>

yes


> * cleanup instructions?
>

no


> ** if anything else, please describe:
>
Does your test system:
> * provide a set of existing tests?
> ** if so, how many?
>
> Not sure I understand the question.


> ==== build management ====
> Does your test system:
> * build the software under test (e.g. the kernel)?
>

yes


> * build the test software?
>

no


> * build other software (such as the distro, libraries, firmware)?
>

no


> * support cross-compilation?
>

yes


> * require a toolchain or build system for the SUT?
>

yes


> * require a toolchain or build system for the test software?
>

yes


> * come with pre-built toolchains?
>

Not published due to repository (github) size limitations, but yes


> * store the build artifacts for generated software?
>

no


> ** in what format is the build metadata stored (e.g. json)?
> ** are the build artifacts stored as raw files or in a database?
> *** if a database, what database?
>
> ==== Test scheduling/management ====
> Does your test system:
> * check that dependencies are met before a test is run?
>

n/a


> * schedule the test for the DUT?
>

DUT questions are n/a: There is no DUT other than qemu, and qemu runs do
not need to be scheduled.


> ** select an appropriate individual DUT based on SUT or test attributes?
> ** reserve the DUT?
> ** release the DUT?
> * install the software under test to the DUT?
> * install required packages before a test is run?
> * require particular bootloader on the DUT? (e.g. grub, uboot, etc.)
> * deploy the test program to the DUT?
> * prepare the test environment on the DUT?
> * start a monitor (another process to collect data) on the DUT?
> * start a monitor on external equipment?
> * initiate the test on the DUT?
> * clean up the test environment on the DUT?
>
> ==== DUT control ====
> Does your test system:
> * store board configuration data?
> ** in what format?
> * store external equipment configuration data?
> ** in what format?
> * power cycle the DUT?
> * monitor the power usage during a run?
> * gather a kernel trace during a run?
> * claim other hardware resources or machines (other than the DUT) for use
> during a test?
> * reserve a board for interactive use (ie remove it from automated
> testing)?
> * provide a web-based control interface for the lab?
> * provide a CLI control interface for the lab?
>
> n/a: There is no DUT other than qemu.


> ==== Run artifact handling ====
> Does your test system:
> * store run artifacts
>

yes


> ** in what format?
>

database


> * put the run meta-data in a database?
>
yes


> ** if so, which database?
>

the database used by buildbot.


> * parse the test logs for results?
>

yes


> * convert data from test logs into a unified format?
>

no


> ** if so, what is the format?
> * evaluate pass criteria for a test (e.g. ignored results, counts or
> thresholds)?
>

yes


> * do you have a common set of result names: (e.g. pass, fail, skip, etc.)
> ** if so, what are they?
>

per buildbot: SUCCESS,WARNINGS,FAILURE,EXCEPTION,RETRY,SKIPPED


> * How is run data collected from the DUT?
> ** e.g. by pushing from the DUT, or pulling from a server?
> * How is run data collected from external equipment?
> * Is external equipment data parsed?
>
> n/a


> ==== User interface ====
> Does your test system:
> * have a visualization system?
>

no


> * show build artifacts to users?
>

no


> * show run artifacts to users?
>

yes


> * do you have a common set of result colors?
>

yes


> ** if so, what are they?
>

per buildbot:
green - passed
orange - warnings
red - failed
blue - aborted


> * generate reports for test runs?
>

no


> * notify users of test results by e-mail?
>
> no


> * can you query (aggregate and filter) the build meta-data?
>

no


> * can you query (aggregate and filter) the run meta-data?
>

no


>
> * what language or data format is used for online results presentation?
> (e.g. HTML, Javascript, xml, etc.)
>

html


> * what language or data format is used for reports? (e.g. PDF, excel, etc.)
>
> n/a


> * does your test system have a CLI control tool?
>

no


> ** what is it called?
>
> ==== Languages: ====
> Examples: json, python, yaml, C, javascript, etc.
> * what is the base language of your test framework core?
>
> python


> What languages or data formats is the user required to learn?
> (as opposed to those used internally)
>
> ==== Can a user do the following with your test framework: ====
> * manually request that a test be executed (independent of a CI trigger)?
>

yes


> * see the results of recent tests?
>

yes


> * set the pass criteria for a test?
>

no


> ** set the threshold value for a benchmark test?
>

no


> ** set the list of testcase results to ignore?
>

no


> * provide a rating for a test? (e.g. give it 4 stars out of 5)
>

no


> * customize a test?
>
no


> ** alter the command line for the test program?
>

no


> ** alter the environment of the test program?
>

no


> ** specify to skip a testcase?
> ** set a new expected value for a test?
> ** edit the test program source?
> * customize the notification criteria?
> ** customize the notification mechanism (eg. e-mail, text)
> * generate a custom report for a set of runs?
> * save the report parameters to generate the same report in the future?
>
> all no


> ==== Requirements ====
> Does your test framework:
> * require minimum software on the DUT?
>

n/a


> * require minimum hardware on the DUT (e.g. memory)
>

n/a


> ** If so, what? (e.g. POSIX shell or some other interpreter, specific
> libraries, command line tools, etc.)
> * require agent software on the DUT? (e.g. extra software besides
> production software)
> ** If so, what agent?
> * is there optional agent software or libraries for the DUT?
> * require external hardware in your labs?
>
> all n/a


> ==== APIS ====
> Does your test framework:
> * use existing APIs or data formats to interact within itself, or with
> 3rd-party modules?
>

no


> * have a published API for any of its sub-module interactions (any of the
> lines in the diagram)?
>

no


> ** Please provide a link or links to the APIs?
>
>
Sorry - this is kind of open-ended...
> * What is the nature of the APIs you currently use?
> Are they:
> ** RPCs?
> ** Unix-style? (command line invocation, while grabbing sub-tool output)
> ** compiled libraries?
> ** interpreter modules or libraries?
> ** web-based APIs?
> ** something else?
>
> Whatever buildbot uses


> ==== Relationship to other software: ====
> * what major components does your test framework use (e.g. Jenkins, Mondo
> DB, Squad, Lava, etc.)
>

buildbot, and its dependencies.


   - Buildbot: 0.8.14
   - Twisted: 18.7.0
   - Jinja: 2.10
   - Python: 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609]



> * does your test framework interoperate with other test frameworks or
> software?
>

No.


> ** which ones?
>
> == Overview ==
> Please list the major components of your test system.
>
> Per buildbot manual.


> {{{Just as an example, Fuego can probably be divided into 3 main parts,
> with somewhat overlapping roles:
> * Jenkins - job triggers, test scheduling, visualization, notification
> * Core - test management (test build, test deploy, test execution, log
> retrieval)
> * Parser - log conversion to unified format, artifact storage, results
> analysis
> There are lots details omitted, but you get the idea.
> }}}
>
> Please list your major components here:
> *
>
> == Glossary ==
> Here is a glossary of terms.  Please indicate if your system uses
> different terms for these concepts.
> Also, please suggest any terms or concepts that are missing.
>
> * Bisection - automatic testing of SUT variations to find the source of a
> problem
>
* Boot - to start the DUT from an off state
> * Build artifact - item created during build of the software under test
> * Build manager (build server) - a machine that performs builds of the
> software under test
> * Dependency - indicates a pre-requisite that must be filled in order for
> a test to run (e.g. must have root access, must have 100 meg of memory,
> some program must be installed, etc.)
> * Device under test (DUT) - the hardware or product being tested (consists
> of hardware under test and software under test) (also 'board', 'target')
> * Deploy - put the test program or SUT on the DUT
> ** this one is ambiguous - some people use this to refer to SUT
> installation, and others to test installation
> * Device under Test (DUT) - a product, board or device that is being tested
> * DUT controller - program and hardware for controlling a DUT (reboot,
> provision, etc.)
> * DUT scheduler - program for managing access to a DUT (take
> online/offline, make available for interactive use)
> ** This is not shown in the CI Loop diagram - it could be the same as the
> Test Scheduler
> * Lab - a collection of resources for testing one or more DUTs (also
> 'board farm')
> * Log - one of the run artifacts - output from the test program or test
> framework
> * Log Parsing - extracting information from a log into a
> machine-processable format (possibly into a common format)
> * Monitor - a program or process to watch some attribute (e.g. power)
> while the test is running
> ** This can be on or off the DUT.
> * Notification - communication based on results of test (triggered by
> results and including results)
> * Pass criteria - set of constraints indicating pass/fail conditions for a
> test
> * Provision (verb) - arrange the DUT and the lab environment (including
> other external hardware) for a test
> ** This may include installing the SUT to the device under test and
> booting the DUT.
> * Report generation - generation of run data into a formatted output
> * Request (noun) - a request to execute a test
> * Result - the status indicated by a test - pass/fail (or something else)
> for a Run
> * Results query - Selection and filtering of data from runs, to find
> patterns
> * Run (noun) - an execution instance of a test (in Jenkins, a build)
> * Run artifact - item created during a run of the test program
> * Serial console - the Linux console connected over a serial connection
> * Software under test (SUT) - the software being tested
> * Test agent - software running on the DUT that assists in test operations
> (e.g. test deployment, execution, log gathering, debugging
> ** One example would be 'adb', for Android-based systems)
> * Test definition - meta-data and software that comprise a particular test
> * Test program - a script or binary on the DUT that performs the test
> * Test scheduler - program for scheduling tests (selecting a DUT for a
> test, reserving it, releasing it)
> * Test software - source and/or binary that implements the test
> * Transport (noun) - the method of communicating and transferring data
> between the test system and the DUT
> * Trigger (noun) - an event that causes the CI loop to start
> * Variant - arguments or data that affect the execution and output of a
> test (e.g. test program command line; Fuego calls this a 'spec')
> * Visualization - allowing the viewing of test artifacts, in aggregated
> form (e.g. multiple runs plotted in a single diagram)
>
> Thank you so much for your assistance in answering this survey!
>
> Regards,
>  -- Tim
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/automated-testing/attachments/20181009/9200fae6/attachment-0001.html>