[Automated-testing] Test Stack Survey

[Kevin Hilman]

<Tim.Bird at sony.com> writes:

>> -----Original Message-----
>> From: Jan Lübbe 
>> 
>> Hi Tim and Kevin,
>> 
>> below are the survey answers for labgrid.
>> 
>> Regarding the diagram, one aspect of labgrid that is not represented
>> there is the interactive access to boards in the lab during development
>> (build a system image locally, login&debug, run a testsuite from the
>> commandline). But with the focus on automated testing, that's probably
>> fine. :)
>
> OK - noted on the wiki page I created.  I think it's worth considering, because
> although it's not directly used in automated testing, it does affect the
> architecture, IMHO.  And I believe some other systems support this.
>
> I created this wiki page for the survey response:
> https://elinux.org/Labgrid_survey_response
>
>> 
> ...[answers omitted]
>
> I have to say I followed your response answers very easily.  Thanks!
> (Maybe that's because I already had some exposure to labgrid)
>
>> == Overview ==
>> Please list the major components of your test system.
>> 
>> Please list your major components here:
>> * exporter: provides access to the DUT and additional test HW
>> * coordinator: monitors exported resources and stores lab
>> configuration/state
>> * client: CLI to control a lab and DUTs
>> * python API: used by pytest testsuites to access the DUTs
>> 
>> == Glossary ==
>> Here is a glossary of terms.  Please indicate if your system uses different
>> terms for these concepts.
>> Also, please suggest any terms or concepts that are missing.
>> 
>> * Dependency - indicates a pre-requisite that must be filled in order for a test
>> to run (e.g. must have root access, must have 100 meg of memory, some
>> program must be installed, etc.)
>>   '''These are called "features" in labgrid'''
>> 
>> '''Missing is the concept of a collection of similar DUTs, as used when
>> scheduling a test on one of many DUTs.'''
>
> Noted.  Do you have a term for this that you use?  I would suggest 'DUT pool', but
> that's just off the top of my head.  I think other systems have similar notions:
> a collection of DUTs with similar features any of which could be used to
> execute a particular test (or which could be used to pipeline multiple tests
> against the same SUT to save time.)
>
> I think it would be good to have a common term for this.

The LAVA terminolgy might be useful here (and already familiar to
some).  In LAVA this is called a "device type", referring to a specific
class/group of similar/identical boards.  A specific instance is called
a "device" in LAVA.

Kevin