[Automated-testing] Test Stack Survey

[Milosz Wasilewski]

On Wed, 3 Oct 2018 at 20:59, <Tim.Bird at sony.com> wrote:
>
> > -----Original Message-----
> > From: Matt Hart
> > Answers for LAVA,
> >
> Thanks for filling this out.  I have a few questions below.
> (I'll omit some lines that I don't have questions about)
>
> I created a page for this response at:
> https://elinux.org/LAVA_survey_response
>
> Note that I'm marking a few things that I think are interesting
> (and may be good to discuss at the summit) in red.
>
> >
> > On Wed, 19 Sep 2018 at 07:39, <Tim.Bird at sony.com> wrote:
> > > ==== test definitions ====
> > > Does your test system:
> > > * have a test definition repository?
> >
> > LAVA does not come with tests, however Linaro does maintain a set of
> > job definitions
>
> Does Linaro make these available for LAVA users?  Are they generalized
> for common usage, or do they apply only to Linaro test projects or labs?
> Where could I find out more about these?

I guess I should answer as co-maintainer of this repository :)
Repo is located here: https://git.linaro.org/qa/test-definitions.git
Tests can be run standalone (as test scripts on DUT) or in LAVA. There
is a README: https://git.linaro.org/qa/test-definitions.git/tree/automated/README
There is also a standalone executor called test-runner.py:
https://git.linaro.org/qa/test-definitions.git/tree/automated/utils/test-runner.py

The main idea is to have a standard way of starting/executing tests
and unified way of reporting results.

Best Regards,
milosz

>
> > > ** if so, what data format or language is used (e.g. yaml, json, shell script)
> >
> > YAML
>
> ...
> > > ==== build management ====
> > > Does your test system:
> > > * build the software under test (e.g. the kernel)?
> >
> > No
> >
> > > * build the test software?
> >
> > Sometimes, quite a lot of LAVA users will build the test software on
> > the device they are testing before executing it
>
> I'm familiar with test systems that rely on the distro build system (e.g. Yocto, Android
> or buildroot) to build the test software and include it in the distro.  But this sounds
> like something different.  Just to verify that I understand - the test program is built
> on the DUT itself?  Is that right?
>
> Is this done: 1) during a test, 2) just before the test as part of test setup, 3) as part
> of DUT provisioning, or 4) some other time?
>
> ...
> > > * store the build artifacts for generated software?
> >
> > No, however pushing to external storage is supported (Artifactorial)
>
> I never heard of this one.  It looks interesting.
> I presume it is this?: https://github.com/ivoire/Artifactorial
>
> ...
> > > ==== Requirements ====
> > > Does your test framework:
> > > * require minimum software on the DUT?
> >
> > Bootloader
> >
> > > * require minimum hardware on the DUT (e.g. memory)
> >
> > Serial port
> >
> > > ** If so, what? (e.g. POSIX shell or some other interpreter, specific
> > libraries, command line tools, etc.)
> >
> > POSIX shell for most DUT, however IOT devices are supported without a shell
>
> Interesting.  I assume this precludes any "interactive" commands, or DUT-side
> operations initiated by LAVA.  (But please correct me if I'm wrong).
> Does LAVA just scan the serial console output for its results tokens?
> I presume that in this circumstance the test initiation is baked into
> the boot sequence for the DUT/SUT?
>
> > > ==== APIS ====
> > > Does your test framework:
> > > * use existing APIs or data formats to interact within itself, or with 3rd-
> > party modules?
> >
> > ZMQ
> >
> > > * have a published API for any of its sub-module interactions (any of the
> > lines in the diagram)?
> > > ** Please provide a link or links to the APIs?
> >
> > No
> >
> > >
> > > Sorry - this is kind of open-ended...
> > > * What is the nature of the APIs you currently use?
> > > Are they:
> > > ** RPCs?
> > > ** Unix-style? (command line invocation, while grabbing sub-tool output)
> > > ** compiled libraries?
> > > ** interpreter modules or libraries?
> > > ** web-based APIs?
> > > ** something else?
> >
> > ZMQ is used between LAVA workers (dispatchers) and the master, to
> > schedule jobs
> > XML-RPC is for users to submit jobs, access results, and control the lab
> >
> > >
> > > ==== Relationship to other software: ====
> > > * what major components does your test framework use (e.g. Jenkins,
> > Mondo DB, Squad, Lava, etc.)
> >
> > Jenkins, Squad
> >
> > > * does your test framework interoperate with other test frameworks or
> > software?
> > > ** which ones?
> >
> > A common LAVA setup is Jenkins to create the builds, LAVA to execute
> > the tests, and Squad/KernelCI to consume the results
>
> That is helpful to know.  Thanks.
>
>
> > >
> > > == Overview ==
> > > Please list the major components of your test system.
> > >
> > > Please list your major components here:
> > > *
> >
> > * LAVA Server - (DUT Scheduler) - UI, Results storage, Device
> > configuration files, Job scheduling, User interaction
> > * LAVA Dispatcher - (DUT Controller) - Device interaction (deploy,
> > boot, execute tests), Device Power Control, Test results Parsing
> >
> > >
> > > == Glossary ==
> > > Here is a glossary of terms.  Please indicate if your system uses different
> > terms for these concepts.
> > > Also, please suggest any terms or concepts that are missing.
> >
> > PDU - Power Distribution Unit, however has become a standard term for
> > automation power control.
>
> Good addition.  Thanks.
>  -- Tim
>
> --
> _______________________________________________
> automated-testing mailing list
> automated-testing at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/automated-testing