[Automated-testing] Farming together - areas of collobration

[Neil Williams]

On 13 February 2018 at 16:50, Steve McIntyre <steve.mcintyre at linaro.org>
wrote:

> On Mon, Feb 12, 2018 at 09:51:37PM +0000, Tim.Bird at sony.com wrote:
> >> From: Steve McIntyre on Monday, February 12, 2018 10:33 AM
> >> On Tue, Nov 14, 2017 at 09:16:01PM +0000, Bird, Timothy wrote:
> >> >
> >> >To be a bit more blunt about it, as a concrete example: would LAVA
> >> >adopt a new system for low-level board control, if it presented
> >> >itself?  I kind of doubt it. Who would do this work?
> >>
> >> That's the big question, yes. In the LAVA team, we've already solved a
> >> lot of the issues that we've seen, in our own ways. We want to be good
> >> Open Source citizens (of course!), but it's difficult to justify
> >> spending much engineering time on ripping things out and moving to
> >> different underpinnings unless we can see concrete benefit. I'd expect
> >> most teams to be the same. Chicken and egg, as you said earlier. The
> >> key thing to make it all worthwhile will be to show the value of doing
> >> it, while minimising the cost. Let's see where we can take that.
>
> ,,,
>
> >One thing I've observed is that there are substantial differences
> >in the level of abstraction and the board management architecture
> >for these different systems.


>From a LAVA perspective, we set the abstraction at:

* Jinja2 for device configuration
* portable shell scripts to run on the DUT or in a container to control the
DUT
* Keep reboot and serial connection support *out* of the test operations
* Retain only generic / relatively basic result presentation / notification
but provide a series of APIs which can be used to present the results in
ways which are suitable to the consumers of the CI lop.

Reboot is a particular problem. Many DUTs require complex steps to get back
into a test environment after a reboot. (In the case of TFTP, it means a
whole new deployment and that's often best done as a separate test job.)
Many existing test frameworks are written to run on a single developer -
single device on my desk model. That then leads to the automation of the
test itself including commands to control the device. It is much more
flexible to isolate the DUT control from the test framework.

I previously had hoped we could do a simple
> >survey of the different systems, and come up with a set of verbs
> >that everyone could use - making it possible for, say, tests written
> >in LAVA to run in a farm that used labgrid or r4d.  I would certainly like
> >to easily run Fuego tests in LAVA-based farms.  Actually, I'd like to make
> >Fuego independent of the board farm management software, so the
> >user can choose.
>
> Right. :-)
>
> >However, it has turned out to not be that simple.  Different systems
> >impose different hardware requirements, or make assumptions about
> >the nature of system deployment, that seem incompatible with
> >other systems.  Or, at least that's my understanding.
> >
> >I'm not sure where to start, but maybe it would be good to have a
> >discussion about the basic requirements and operation of each
> >system, to see where the compatibilities and incompatibilities are.
> >(Or, maybe start with the requirements that the upper-level
> >software or user have, that are using the different systems).
>
> Nod, that sounds like the best place to start. Get people in a room
> and swap what we know and expect.
>
> >Would it be worth putting together some kind of "board farm summit"
> >at the next plumbers?  I would suggest ELC, but it's already programmed,
> >and coming up soon, and Linaro has Connect going on the same month, so
> >I expect travel would be an issue.
>
> Plumbers sounds like a good plan for me and other Linaro
> folks. Looking at the ELC dates, it's not *quite* clashing with our
> next Connect in Hong Kong, but it's not likely to work with only a few
> days between.
>
> Cheers,
> --
> Steve McIntyre                                steve.mcintyre at linaro.org
> <http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
>
> --
> _______________________________________________
> automated-testing mailing list
> automated-testing at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/automated-testing
>



-- 

Neil Williams
=============
neil.williams at linaro.org
http://www.linux.codehelp.co.uk/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/automated-testing/attachments/20180219/7ede2b6b/attachment-0001.html>