[Automated-testing] CKI Test Stack Survey (was RE: CKI project introduction)

Veronika Kabatova vkabatov at redhat.com
Thu Mar 28 05:38:56 PDT 2019



----- Original Message -----
> From: "Veronika Kabatova" <vkabatov at redhat.com>
> To: "Tim Bird" <Tim.Bird at sony.com>
> Cc: automated-testing at yoctoproject.org
> Sent: Thursday, March 28, 2019 12:41:44 PM
> Subject: Re: [Automated-testing] CKI Test Stack Survey (was RE: CKI project	introduction)
> 
> 

I left our some more information that might be useful...

> 
> ----- Original Message -----
> > From: "Tim Bird" <Tim.Bird at sony.com>
> > To: vkabatov at redhat.com
> > Cc: automated-testing at yoctoproject.org
> > Sent: Wednesday, March 27, 2019 11:29:39 PM
> > Subject: CKI Test Stack Survey (was RE: CKI project introduction)
> > 
> > > -----Original Message-----
> > > From: Veronika Kabatova
> > > 
> > > ----- Original Message -----
> > > > From: "Tim Bird" <Tim.Bird at sony.com>
> > ...
> > > > We would like to learn more about your system, to see how it fits with
> > > > our
> > > > conception of
> > > > the CI loop that we use in embedded Linux testing.   To help use share
> > > > information
> > > > among ourselves, we created a survey, which I can send to you if you
> > > > are
> > > > willing
> > > > to fill it out.  You can see the survey and responses for other Linux
> > > > test
> > > > systems at:
> > > > https://elinux.org/Test_Stack_Survey
> > > >
> > > 
> > > Absolutely! Should I make a page with our answers on the web or do you
> > > have a different way of recording them?
> > 
> > It's handy if you provide the answers in wiki format, but instead of
> > putting
> > it on the wiki directly, I prefer if you send the response via e-mail
> > (to me, with <automated-testing at yoctoproject.org> CC'ed)
> > That way, I review it when I add it to the wiki myself, and we can ask
> > questions about your stack on the mailing list easily.
> > 
> > Here is the survey.  Sorry it's so long, but many questions should be easy
> > to
> > answer.
> > Please refer to the attached diagram, or see the page:
> > https://elinux.org/Test_Stack_Survey.
> > 
> > The format of the survey is mediawiki markup, so * = bullet, ** = indented
> > bullet.
> > 
> > == Diagrams ==
> > Attached is a diagram for the high level CI loop:
> > 
> > The boxes represent different processes, hardware, or storage locations.
> > Lines between boxes indicate APIs or control flow,
> > and are labeled with letters.  The intent of this is to facilitate
> > discussion
> > at the summit.
> > 
> > [[File:<see attachment>|high level CI loop]]
> > --------
> > 
> > If you have an element that is not featured in this diagram, please let us
> > know.
> > 
> 
> Te diagram more or less works, there are just small differences in some of
> our
> pipelines (eg. we grab completed koji builds for testing Fedora kernels so in
> that case we don't touch the source nor build the kernel). But for generic
> purposes it works and it is indeed very similar to our pipelines for source
> testing.
> 
> (I removed the quotes below so you have easier time copying over the whole
> text)
> 
> 
> == Cover text ==
> Hello Test Framework developer or user,
> 
> The purpose of this survey is to try to understand how different Test
> Frameworks and Automated Test components
> in the Linux Test ecosystem work - what features they have, what terminology
> they use, and so forth.
> The reason to characterize these different pieces of software (and hardware)
> is to try to come up
> with definitions for a Test Stack, and possibly API definitions, that will
> allow different
> elements to communicate and interact.  We are interested in seeing the
> commonalities and differences
> between stack elements.
> 
> This information was used, to start, to prepare for discussions about test
> stack standards
> at the Automated Testing Summit 2018.
> 
> Please see the Glossary below for the meaning of words used in this survey.
> If you use different
> words in your framework for the same concept, please let us know.  If you
> think there are other
> words that should be in the Glossary, please let us know.
> 
> == Survey Questions ==
> * What is the name of your test framework? '''CKI Project (Continuous Kernel
> Integration)'''
> 
> Which of the aspects below of the CI loop does your test framework perform?
> 
> The answers can be: "yes", "no", or "provided by the user".
> Where the answer is not simply yes or no, an explanation is appreciated.
> 
> For example, in Fuego, Jenkins is used for trigger detection (that is, to
> detect new SUT versions), but the user must install the Jenkins module and
> configure this themselves.  So Fuego supports triggers, but does not provide
> them pre-configured for the user.
> 
> If the feature is provided by a named component in your system (or by an
> external module), please provide the name of that module.
> 
> Does your test framework:
> ==== source code access ====
> * access source code repositories for the software under test?
> '''Yes (unless testing RPM builds)'''
> * access source code repositories for the test software?
> '''Yes'''
> * include the source for the test software?
> '''Yes'''
> * provide interfaces for developers to perform code reviews?
> '''No'''
> * detect that the software under test has a new version?
> '''Yes'''
> ** if so, how? (e.g. polling a repository, a git hook, scanning a mail list,
> etc.)
> '''Polling git trees and Patchwork, listening on fedmsg bus'''
> * detect that the test software has a new version?
> '''Yes'''
> 
> ==== test definitions ====
> Does your test system:
> * have a test definition repository?
> '''Yes, multiple'''
> ** if so, what data format or language is used (e.g. yaml, json, shell
> script)
> '''For metadata YAML. Tests themselves can be any language if they are
> wrapped as [https://beaker-project.org/ Beaker tasks], which requires some
> XML snippets.'''
> 
> Does your test definition include:
> * source code (or source code location)?
> '''Yes (part of configuration)'''
> * dependency information?
> '''Yes (each test is responsible for handling its dependencies)'''
> * execution instructions?
> '''Yes (in each test's README)'''
> * command line variants?
> '''Yes (in each test's README)'''
> * environment variants?
> '''Yes'''
> * setup instructions?
> '''Yes (in each test's README)'''
> * cleanup instructions?
> '''No, each test is responsible to clean up after itself'''
> ** if anything else, please describe:
> 
> Does your test system:
> * provide a set of existing tests?
> '''Yes'''
> ** if so, how many?
> '''Currently 24 tests open sourced but there are some more running
> internally.
> Some tests' reliability is a WIP.'''
> 

Some tests are architecture/tree specific so a full test set is not
executed for each run.

> 
> ==== build management ====
> Does your test system:
> * build the software under test (e.g. the kernel)?
> '''Yes, unless testing already build RPMs (eg. Fedora kernels built in koji
> and COPR)'''
> * build the test software? (e.g. dbench)?
> '''Yes, if needed'''
> * build other software (such as the distro, libraries, firmware)?
> '''No, everything is installed by [https://beaker-project.org/ Beaker] from
> images and distro repositories'''
> * support cross-compilation?
> '''Yes'''
> * require a toolchain or build system for the SUT?
> '''Yes'''
> * require a toolchain or build system for the test software?
> '''Test-dependent, some need to be compiled and some not'''
> * come with pre-built toolchains?
> '''Yes, RPMs installed in Beaker before testing starts. Build containers with
> toolchains for kernel compilation are also available at
> [https://gitlab.com/cki-project/cki-containers/]'''
> * store the build artifacts for generated software?
> '''Yes'''
> ** in what format is the build metadata stored (e.g. json)?
> '''GitLab CI's variables and pipelines'''
> ** are the build artifacts stored as raw files or in a database?
> '''Raw files (GitLab artifacts)'''
> *** if a database, what database?
> 
> ==== Test scheduling/management ====
> Does your test system:
> * check that dependencies are met before a test is run?
> '''Yes, each test is responsible for listing its dependencies and aborts
> if they can't be installed'''
> * schedule the test for the DUT?
> '''Yes'''
> ** select an appropriate individual DUT based on SUT or test attributes?
> '''Yes'''
> ** reserve the DUT?
> '''Yes (for the test duration, but can be configured to extend the
> reservation)'''
> ** release the DUT?
> '''Yes'''
> * install the software under test to the DUT?
> '''Yes'''
> * install required packages before a test is run?
> '''Yes'''
> * require particular bootloader on the DUT? (e.g. grub, uboot, etc.)
> '''No'''
> * deploy the test program to the DUT?
> '''Yes'''
> * prepare the test environment on the DUT?
> '''Yes'''
> * start a monitor (another process to collect data) on the DUT?
> '''Yes, part of Beaker's framework'''
> * start a monitor on external equipment?
> '''It's part of Beaker labs but we don't use it'''
> * initiate the test on the DUT?
> '''Yes'''
> * clean up the test environment on the DUT?
> '''Yes'''
> 
> Most of the above is handled by [https://beaker-project.org/ Beaker]
> 
> 
> ==== DUT control ====
> Does your test system:
> * store board configuration data?
> '''Yes, stored in Beaker'''
> ** in what format?
> 
> No idea, we aren't Beaker developers
> 
> * store external equipment configuration data?
> 
> No idea, we aren't Beaker developers
> 
> ** in what format?
> * power cycle the DUT?
> '''Yes'''
> * monitor the power usage during a run?
> '''No'''
> * gather a kernel trace during a run?
> '''We have the functionality but it's not in use currently.'''
> * claim other hardware resources or machines (other than the DUT) for use
> during a test?
> '''Yes, test dependent'''
> * reserve a board for interactive use (ie remove it from automated testing)?
> '''Yes, configurable (currently disabled for CI runs)'''
> * provide a web-based control interface for the lab?
> '''Yes'''
> * provide a CLI control interface for the lab?
> '''Yes'''
> 
> 
> ==== Run artifact handling ====
> Does your test system:
> * store run artifacts
> '''Yes'''
> ** in what format?
> '''XML/JSON, provided by Beaker'''
> * put the run meta-data in a database?
> 
> Does GitLab's pipeline history count as a DB?
> 
> ** if so, which database?
> * parse the test logs for results?
> '''Only the results, not logs. The onboarded tests need to give clear
> results, otherwise we don't run them'''
> * convert data from test logs into a unified format?
> '''No'''
> ** if so, what is the format?
> * evaluate pass criteria for a test (e.g. ignored results, counts or
> thresholds)?
> '''Yes'''
> * do you have a common set of result names: (e.g. pass, fail, skip, etc.)
> '''Yes'''
> ** if so, what are they?
> '''Provided by Beaker: PASS, FAIL, WARN, SKIP, PANIC, NONE, NEW. These get
> interpreted into binary pass/fail in the reports. We use a WAIVED status on
> top of these for test results that can be ignored.'''
> * How is run data collected from the DUT?
> ** e.g. by pushing from the DUT, or pulling from a server?
> '''Pulled from Beaker'''
> * How is run data collected from external equipment?
> '''N/A'''
> * Is external equipment data parsed?
> '''N/A'''
> 
> 
> ==== User interface ====
> Does your test system:
> * have a visualization system?
> '''No proper dashboard yet but results can be queried in GitLab (only
> available internally right now)'''
> * show build artifacts to users?
> '''Yes, links in reports and available in GitLab'''
> * show run artifacts to users?
> '''No logs, only results. We need to deal with log sanitizing first'''
> * do you have a common set of result colors?
> '''Partially'''
> ** if so, what are they?
> '''Red (failure) vs green (pass) for GitLab pipelines, 'X' vs check-mark
> emoji in reports'''
> * generate reports for test runs?
> '''Yes'''
> * notify users of test results by e-mail?
> '''Yes'''
> 
> * can you query (aggregate and filter) the build meta-data?
> '''Yes but limited (GitLab's pipeline jobs and reports)'''
> * can you query (aggregate and filter) the run meta-data?
> '''Yes but limited (GitLab's pipeline jobs, reports and Beaker results)'''
> 
> * what language or data format is used for online results presentation? (e.g.
> HTML, Javascript, xml, etc.)
> 
> Whatever GitLab WEBUI uses since that's the only thing we have right now.
> 
> * what language or data format is used for reports? (e.g. PDF, excel, etc.)
> '''Plaintext emails'''
> * does your test system have a CLI control tool?
> '''Yes, multiple different tools for different parts of the pipeline'''
> ** what is it called?
> '''bkr (Beaker CLI), set of Python scripts to retrigger and query pipelines,
> [https://github.com/cki-project/kpet kpet] for patch evaluation and test
> picking, [https://github.com/cki-project/skt skt] for patch application,
> kernel building and Beaker interaction... and I'm sure I'm forgetting some
> other ones'''
> 
> 
> ==== Languages: ====
> Examples: json, python, yaml, C, javascript, etc.
> * what is the base language of your test framework core?
> '''Beaker tests use beakerlib/restraint wrappers (Bash/Makefiles) but tests
> themselves can be in any language. CKI itself uses Python and YAML the most,
> with some Bash thrown in'''
> 
> What languages or data formats is the user required to learn?
> (as opposed to those used internally)
> '''YAML'''
> 
> ==== Can a user do the following with your test framework: ====
> 
> '''Please note that we are heavily relying on internal labs and thus
> all potential users and their configuration need to go through us
> first, as the internal infrastructure is not available from public
> networks.'''
> 
> * manually request that a test be executed (independent of a CI trigger)?
> '''No (unless they manually submit a Beaker job)'''
> * see the results of recent tests?
> '''Yes (filter GitLab, Beaker and reports). Limited functionality right
> now'''
> * set the pass criteria for a test?
> '''Yes, file a PR for test modification (granularity of arch/HW/distro/kernel
> version etc. is doable)'''
> ** set the threshold value for a benchmark test?
> '''Yes, see above'''
> ** set the list of testcase results to ignore?
> '''Yes, but the set of ignored results is global for all CI runs right now'''

While the configuration we currently use is global, it can be easily adjusted
for the test to be ignored only on specific arches/kernel trees.

> * provide a rating for a test? (e.g. give it 4 stars out of 5)
> '''Yes, please reply to the report or file an issue for the test'''
> * customize a test?
> '''Yes, file a PR with requested changes and reasoning, or fork a test and
> change the metadata for runs'''
> ** alter the command line for the test program?
> '''Yes, see above'''
> ** alter the environment of the test program?
> '''Yes, see above'''
> ** specify to skip a testcase?
> '''Yes, see above'''
> ** set a new expected value for a test?
> '''Yes, see above'''
> ** edit the test program source?
> '''Yes, see above'''
> * customize the notification criteria?
> '''Partially, we offer turning off "pass" emails if you are interested in
> seeing only failures'''
> ** customize the notification mechanism (eg. e-mail, text)
> '''No, only email available right now'''
> * generate a custom report for a set of runs?
> '''No'''
> * save the report parameters to generate the same report in the future?
> '''Yes, the metadata is saved in the pipeline so the same report can be
> recreated by calling the reporter on the same pipeline'''
> 
> ==== Requirements ====
> Does your test framework:
> * require minimum software on the DUT?
> '''Tree dependent, installed by Beaker before testing starts'''
> * require minimum hardware on the DUT (e.g. memory)
> '''Test dependent, HW chosen based on test (and tree) requirements'''
> ** If so, what? (e.g. POSIX shell or some other interpreter, specific
> libraries, command line tools, etc.)
> '''Most of the toolchains and library versions are decided by the distro
> that's used and that depends on what we are testing. Eg. we can't compile
> upstream kernels with CentOS 7 toolchain'''
> * require agent software on the DUT? (e.g. extra software besides production
> software)
> '''Yes'''
> ** If so, what agent?
> '''restraint/beakerlib (see [https://beaker-project.org/ Beaker website] for
> details)'''
> * is there optional agent software or libraries for the DUT?
> 
> Not sure what this question means?
> 
> * require external hardware in your labs?
> '''No'''
> 
> ==== APIS ====
> Does your test framework:
> * use existing APIs or data formats to interact within itself, or with
> 3rd-party modules?
> '''Yes'''
> * have a published API for any of its sub-module interactions (any of the
> lines in the diagram)?
> '''Yes'''
> ** Please provide a link or links to the APIs?
> ** '''Python's GitLab API:
> [https://python-gitlab.readthedocs.io/en/stable/]'''
> ** '''Beaker job submission XML:
> [https://beaker-project.org/docs/_downloads/beaker-job.rng]'''
> ** '''Patchwork v2 REST API:
> [https://patchwork.readthedocs.io/en/latest/api/rest/]'''
> ** '''Fedmsg receiver for COPR and Koji:
> [https://fedora-fedmsg.readthedocs.io/en/latest/topics.html]'''
> 
> 
> Sorry - this is kind of open-ended...
> * What is the nature of the APIs you currently use?
> Are they:
> ** RPCs?
> ** Unix-style? (command line invocation, while grabbing sub-tool output)
> ** compiled libraries?
> ** interpreter modules or libraries?
> ** web-based APIs?
> ** something else?
> 
> Does the above answer these questions too?
> 
> ==== Relationship to other software: ====
> * what major components does your test framework use (e.g. Jenkins, Mondo DB,
> Squad, Lava, etc.)
> '''GitLab, Beaker, UpShift'''
> * does your test framework interoperate with other test frameworks or
> software?
> '''Yes'''
> ** which ones?
> '''Any tests can be used if they are wrapped into Beaker task'''
> 
> == Overview ==
> Please list the major components of your test system.
> 
> {{{Just as an example, Fuego can probably be divided into 3 main parts, with
> somewhat overlapping roles:
> * Jenkins - job triggers, test scheduling, visualization, notification
> * Core - test management (test build, test deploy, test execution, log
> retrieval)
> * Parser - log conversion to unified format, artifact storage, results
> analysis
> There are lots details omitted, but you get the idea.
> }}}
> 
> Please list your major components here:
> * UpShift containers -- actual pipeline core (GitLab runners), triggers,
> kernel
> compilation
> * Beaker -- machine provisioning and testing
> * GitLab -- The whole pipeline runs inside, contains all metadata and
> configuration
> 
> == Glossary ==
> Here is a glossary of terms.  Please indicate if your system uses different
> terms for these concepts.
> Also, please suggest any terms or concepts that are missing.
> 
> * Bisection - automatic testing of SUT variations to find the source of a
> problem
> * Boot - to start the DUT from an off state
> * Build artifact - item created during build of the software under test
> * Build manager (build server) - a machine that performs builds of the
> software under test
> * Dependency - indicates a pre-requisite that must be filled in order for a
> test to run (e.g. must have root access, must have 100 meg of memory, some
> program must be installed, etc.)
> * Deploy - put the test program or SUT on the DUT
> ** this one is ambiguous - some people use this to refer to SUT installation,
> and others to test installation
> * Device under Test (DUT) - a product, board or device that is being tested
> ** the hardware or product being tested (consists of hardware under test and
> software under test) (also 'board', 'target')
> * DUT controller - program and hardware for controlling a DUT (reboot,
> provision, etc.)
> * DUT scheduler - program for managing access to a DUT (take online/offline,
> make available for interactive use)
> ** This is not shown in the CI Loop diagram - it could be the same as the
> Test Scheduler
> * Lab - a collection of resources for testing one or more DUTs (also 'board
> farm')
> * Log - one of the run artifacts - output from the test program or test
> framework
> * Log Parsing - extracting information from a log into a machine-processable
> format (possibly into a common format)
> * Monitor - a program or process to watch some attribute (e.g. power) while
> the test is running
> ** This can be on or off the DUT.
> * Notification - communication based on results of test (triggered by results
> and including results)
> * Pass criteria - set of constraints indicating pass/fail conditions for a
> test
> * Provision (verb) - arrange the DUT and the lab environment (including other
> external hardware) for a test
> ** This may include installing the SUT to the device under test and booting
> the DUT.
> * Report generation - generation of run data into a formatted output
> * Request (noun) - a request to execute a test
> * Result - the status indicated by a test - pass/fail (or something else) for
> a Run
> * Results query - Selection and filtering of data from runs, to find patterns
> * Run (noun) - an execution instance of a test (in Jenkins, a build)
> * Run artifact - item created during a run of the test program
> * Serial console - the Linux console connected over a serial connection
> * Software under test (SUT) - the software being tested
> * Test agent - software running on the DUT that assists in test operations
> (e.g. test deployment, execution, log gathering, debugging
> ** One example would be 'adb', for Android-based systems)
> * Test definition - meta-data and software that comprise a particular test
> * Test program - a script or binary on the DUT that performs the test
> * Test scheduler - program for scheduling tests (selecting a DUT for a test,
> reserving it, releasing it)
> * Test software - source and/or binary that implements the test
> * Transport (noun) - the method of communicating and transferring data
> between the test system and the DUT
> * Trigger (noun) - an event that causes the CI loop to start
> * Variant - arguments or data that affect the execution and output of a test
> (e.g. test program command line; Fuego calls this a 'spec')
> * Visualization - allowing the viewing of test artifacts, in aggregated form
> (e.g. multiple runs plotted in a single diagram)
> 
> 
> 
> > 
> > Thank you so much for your assistance in answering this survey!
> > 
> 
> I left some questions without answer since I'm not sure about them. Please
> ask for clarification if you need more details. I already promised to write
> up some blog posts with details about our architecture but am not sure when
> they'll be ready.
> 
> 
> Veronika
> 
> > Regards,
> >  -- Tim
> > 
> > 
> --
> _______________________________________________
> automated-testing mailing list
> automated-testing at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/automated-testing
> 


More information about the automated-testing mailing list