[Automated-testing] Test Stack Survey

[Jan Lübbe]

On Wed, 2018-10-03 at 03:00 +0000, Tim.Bird at sony.com wrote:
> > -----Original Message-----
> > From: Jan Lübbe 
> > 
> > Hi Tim and Kevin,
> > 
> > below are the survey answers for labgrid.
> > 
> > Regarding the diagram, one aspect of labgrid that is not represented
> > there is the interactive access to boards in the lab during development
> > (build a system image locally, login&debug, run a testsuite from the
> > commandline). But with the focus on automated testing, that's probably
> > fine. :)
> 
> OK - noted on the wiki page I created.  I think it's worth considering, because
> although it's not directly used in automated testing, it does affect the
> architecture, IMHO.  And I believe some other systems support this.
> 
> I created this wiki page for the survey response:
> https://elinux.org/Labgrid_survey_response

Thanks!

> ...[answers omitted]
> 
> I have to say I followed your response answers very easily.  Thanks!
> (Maybe that's because I already had some exposure to labgrid)

Let's hope that others find them useful as well. :)

> > == Overview ==
> > Please list the major components of your test system.
> > 
> > Please list your major components here:
> > * exporter: provides access to the DUT and additional test HW
> > * coordinator: monitors exported resources and stores lab
> > configuration/state
> > * client: CLI to control a lab and DUTs
> > * python API: used by pytest testsuites to access the DUTs
> > 
> > == Glossary ==
> > Here is a glossary of terms.  Please indicate if your system uses different
> > terms for these concepts.
> > Also, please suggest any terms or concepts that are missing.
> > 
> > * Dependency - indicates a pre-requisite that must be filled in order for a test
> > to run (e.g. must have root access, must have 100 meg of memory, some
> > program must be installed, etc.)
> >   '''These are called "features" in labgrid'''
> > 
> > '''Missing is the concept of a collection of similar DUTs, as used when
> > scheduling a test on one of many DUTs.'''
> 
> Noted.  Do you have a term for this that you use?  I would suggest 'DUT pool', but
> that's just off the top of my head.  I think other systems have similar notions:
> a collection of DUTs with similar features any of which could be used to
> execute a particular test (or which could be used to pipeline multiple tests
> against the same SUT to save time.)
> 
> I think it would be good to have a common term for this.

Note that labgrid doesn't have support for this yet, so we haven't
decided on a name. :) LAVA's 'device type' isn't bad.

We may want to have the ability to put one DUT into several
(overlapping) groups, so having 'labels' like the Jenkins executors may
be a better fit. Otherwise 'device types', 'groups' or 'classes' come
to mind.

> Can you add a pointer to your ELCE presentation on labgrid from last year,
> in the "Additional Data" section of the wiki page for your survey response?

Done.

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |