[Automated-testing] Test Stack Survey

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Matt Hart 
> Answers for LAVA,
> 
Thanks for filling this out.  I have a few questions below.
(I'll omit some lines that I don't have questions about)

I created a page for this response at:
https://elinux.org/LAVA_survey_response

Note that I'm marking a few things that I think are interesting
(and may be good to discuss at the summit) in red.

> 
> On Wed, 19 Sep 2018 at 07:39, <Tim.Bird at sony.com> wrote:
> > ==== test definitions ====
> > Does your test system:
> > * have a test definition repository?
> 
> LAVA does not come with tests, however Linaro does maintain a set of
> job definitions

Does Linaro make these available for LAVA users?  Are they generalized
for common usage, or do they apply only to Linaro test projects or labs?
Where could I find out more about these?
 
> > ** if so, what data format or language is used (e.g. yaml, json, shell script)
> 
> YAML

...
> > ==== build management ====
> > Does your test system:
> > * build the software under test (e.g. the kernel)?
> 
> No
> 
> > * build the test software?
> 
> Sometimes, quite a lot of LAVA users will build the test software on
> the device they are testing before executing it

I'm familiar with test systems that rely on the distro build system (e.g. Yocto, Android
or buildroot) to build the test software and include it in the distro.  But this sounds
like something different.  Just to verify that I understand - the test program is built
on the DUT itself?  Is that right?

Is this done: 1) during a test, 2) just before the test as part of test setup, 3) as part
of DUT provisioning, or 4) some other time?

...
> > * store the build artifacts for generated software?
> 
> No, however pushing to external storage is supported (Artifactorial)

I never heard of this one.  It looks interesting.
I presume it is this?: https://github.com/ivoire/Artifactorial

...
> > ==== Requirements ====
> > Does your test framework:
> > * require minimum software on the DUT?
> 
> Bootloader
> 
> > * require minimum hardware on the DUT (e.g. memory)
> 
> Serial port
> 
> > ** If so, what? (e.g. POSIX shell or some other interpreter, specific
> libraries, command line tools, etc.)
> 
> POSIX shell for most DUT, however IOT devices are supported without a shell

Interesting.  I assume this precludes any "interactive" commands, or DUT-side
operations initiated by LAVA.  (But please correct me if I'm wrong).
Does LAVA just scan the serial console output for its results tokens?
I presume that in this circumstance the test initiation is baked into
the boot sequence for the DUT/SUT?
 
> > ==== APIS ====
> > Does your test framework:
> > * use existing APIs or data formats to interact within itself, or with 3rd-
> party modules?
> 
> ZMQ
> 
> > * have a published API for any of its sub-module interactions (any of the
> lines in the diagram)?
> > ** Please provide a link or links to the APIs?
> 
> No
> 
> >
> > Sorry - this is kind of open-ended...
> > * What is the nature of the APIs you currently use?
> > Are they:
> > ** RPCs?
> > ** Unix-style? (command line invocation, while grabbing sub-tool output)
> > ** compiled libraries?
> > ** interpreter modules or libraries?
> > ** web-based APIs?
> > ** something else?
> 
> ZMQ is used between LAVA workers (dispatchers) and the master, to
> schedule jobs
> XML-RPC is for users to submit jobs, access results, and control the lab
> 
> >
> > ==== Relationship to other software: ====
> > * what major components does your test framework use (e.g. Jenkins,
> Mondo DB, Squad, Lava, etc.)
> 
> Jenkins, Squad
> 
> > * does your test framework interoperate with other test frameworks or
> software?
> > ** which ones?
> 
> A common LAVA setup is Jenkins to create the builds, LAVA to execute
> the tests, and Squad/KernelCI to consume the results

That is helpful to know.  Thanks.

 
> >
> > == Overview ==
> > Please list the major components of your test system.
> >
> > Please list your major components here:
> > *
> 
> * LAVA Server - (DUT Scheduler) - UI, Results storage, Device
> configuration files, Job scheduling, User interaction
> * LAVA Dispatcher - (DUT Controller) - Device interaction (deploy,
> boot, execute tests), Device Power Control, Test results Parsing
> 
> >
> > == Glossary ==
> > Here is a glossary of terms.  Please indicate if your system uses different
> terms for these concepts.
> > Also, please suggest any terms or concepts that are missing.
> 
> PDU - Power Distribution Unit, however has become a standard term for
> automation power control.

Good addition.  Thanks.
 -- Tim