[Automated-testing] Farming together - areas of collobration

[Steve McIntyre]

Hey Bill, nice to see even more familiar folks here... :-)

On Mon, Feb 19, 2018 at 01:13:00PM -0500, William Mills wrote:
>On 02/19/2018 03:39 AM, Neil Williams wrote:
>> 
>> Reboot is a particular problem. Many DUTs require complex steps to get
>> back into a test environment after a reboot. (In the case of TFTP, it
>> means a whole new deployment and that's often best done as a separate
>> test job.) Many existing test frameworks are written to run on a single
>> developer - single device on my desk model. That then leads to the
>> automation of the test itself including commands to control the device.
>> It is much more flexible to isolate the DUT control from the test framework.
>
>There are actually multiple concerns here:
>
>1) Isolation
>
>A test should not be able to interfere with devices not given to it as a
>resource.  (Ex: A test on DUT1 should not be able to reset DUT2 unless
>DUT2 is also part of the same test).
>
>2) Abstraction
>
>Tests that do not care how the device gets booted should not have code
>in them that does that.  This makes tesst portable and maintainable.
>
>3) Boot Control
>
>A test should be able to control the details of boot and deployment if
>it needs to.
>
>--------
>
>I actually think both LAVA dispatcher and VATF (TI Opentest equivalent)
>have this wrong today.  LAVA achieves 1 & 2.  VATF achieves 2 & 3.
>
>VATF today fails at #1 because it has equipment drivers that directly
>control the PDU, relays etc.  This is a problem for isolation as it
>allows a badly written test case to mess with other independent test cases.

ACK.

>It is not a failure of abstraction because power & relay control (among
>others) are abstracted in the code base so test benches can have
>different methods of power control w/o changing the test case logic.
>
>New equipment drivers could be written for VATF to talk to labgrid or
>other daemon that will allow it to control only the resources (PDU
>sockets, serial ports, relays, Digital Power Meter) that were assigned
>by the scheduler.
>
>This would fix #1-Isolation for VATF and very few if any test cases
>would need to be re-written because of power, relay, & serial port
>abstraction levels.
>
>We agree that #2-Abstraction is very important.  However having the
>*capability* to control the low level details does not cause a violation
>of this principal.  The vast majority of VATF test cases do not have any
>logic on how to boot the board.  Instead they inherit the default "boot"
>method.  The default boot method will look at the board type and the
>capabilities & policies of the test bench and do the right thing.  This
>can be enhanced via test run arguments if you wish (run the same test
>via different deployment models for example)
>
>#3-Control is important to us and others.  With low level control we can
>test things like:
>* Do 1000 boot cycles with pseudo random off times between 0.5 and 2
>seconds and categorize the results
>* Test the SOC ROMs serial boot method
>* Test the SOC ROMs TFTP boot method
>* Test u-boots TFTP method
>* Test u-boots USB boot method
>
>These tests are indeed less portable than the ones for case #2.  The SOC
>ROM serial boot test cases would be TI specific and probably needs a
>tweak for each new TI SOC.  However it is test bench portable.  It just
>needs a DUT that has power control, serial port connection, and bootmode
>pin control.  The test case does not care if the bootmode is controlled
>via 1 relay or 8 relays or which manufacture those relays come from.
>
>The SOC ROM test case would declare that it needs bootmode control (or
>perhaps it declares which bootmodes it needs). The scheduler would only
>run that test on boards that had the bootmode control needed.  (Bootmode
>control would not be needed on all boards, just a subset.)

OK. This is where we've diverged in terms of design, then, in a couple
of significant ways. That's not a criticism in any way, but more an
acknowledgement of different experiences and requirements during
development of LAVA and VATF.

The primary targets for early versions of LAVA used to be much
simpler: start with a Pandaboard, either put an image on the SD card
using SDmux and boot it, or drive the existing U-Boot and TFTP boot an
image. We were ~always concentrating testing the higher levels of the
stack than U-Boot, unlike your description in #3 above.

Since then, we've seen more and more complex, *awkward* boards that
need significantly more hand-holding to get things to boot at all.

 * In terms of isolation (your point #1), tools like adb and fastboot
   are awfully keen on attempting to interrogate and interact with all
   the devices they can find on the USB bus on your server (as Neil
   alluded to above in the single developer - single device
   model). They work fine if you have only one DUT attached to your
   test server, but can screw things up utterly if you're trying to
   run multiple independent tests on multiple DUTs in parallel.

 * Depending on the device, it's possible to choose at boot time
   whether you want to boot an Android-style image (split into
   multiple inter-dependent partitions) or switch to a UEFI loader and
   go onto a more generic Linux image like OE. We've had requirements
   to support both options on the same devices in the lab, which leads
   to a lot of hard work to make that possible. It's also ~impossible
   to support tests running on these DUTs rebooting themselves
   directly; they have very little chance of reliably booting back
   into the same image they just came from.

As we've spent more and more time supporting that kind of device in
LAVA, that's coloured our design decisions. As people were often
wanting to run more generic tests on top of the DUTs available, we've
also ended up trying (as much as possible) to separate the details of
deploying to and (re)booting a DUT from the tests specified. A lot of
the logic for how to deal with the awkward boards has ended up
embedded in the lower levels of LAVA, which makes it easier for us to
abstract some of the details away.

At the moment, LAVA has many boot methods or combinations of boot
methods available to each DUT and many DUTs support several
combinations. LAVA doesn't restrict how the components are used, and
the total number of permutations isn't obvious. So a default U-Boot
method makes no sense for such use cases. We have base templates for
U-Boot and fastboot, amongst others. base-uboot is based on TFTP with
no permanent storage, so rebooting the device in the test environment
would simply land at a U-Boot> prompt, not into a booted system.

<snip>

>BTW: I have talked about VATF a lot above.
>It is TODO to write up / update our OpenTest info.  Here is some quick info:
>
>http://arago-project.org/wiki/index.php/Opentest

ACK - looks interesting. More details would be lovely. :-)

>OpenTest was created by TI in 2009 and is used primarily by TI.
>It uses existing open source projects (TestLink and STAF) and adds new
>content from TI. The TI framework and test case logic are open source
>(BSD) but the test database is closed (for now anyway).
>
>It was made open source largely so our customers could duplicate our
>test environment. So far very few have done so but we will continue to
>work on it as open source.
>
>We have a very small team that works on it so we are not scaled for wide
>spread deployment and community adoption.  We think there is a lot of
>good principals in it.  Please borrow good ideas from it at will and
>please don't copy the mistakes (like the use of Ruby :) ).

*grin* Every time someone chooses a programming language, the choice
quickly becomes permanent...

>We plan to continue to use OpenTest as our main test framework and VATF
>as a major "Test Execution Engine".  However, OpenTest allows multiple
>TEEs to be used on the same board for different tests so we have
>integrated LAVA dispatcher as a TEE also.

Aha! Interesting to know!

Cheers,
-- 
Steve McIntyre                                steve.mcintyre at linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs