[Automated-testing] Farming together - areas of collobration

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Steve McIntyre on Monday, February 12, 2018 10:33 AM
> On Tue, Nov 14, 2017 at 09:16:01PM +0000, Bird, Timothy wrote:
> 
> >I think one of the most difficult things will be building the ecosystem to
> >support a solution here.  There are issues with finding a home for this
> >stuff, in that someone will have to do maintenance work and support use
> cases
> >that don't apply to themselves.  In other words, I'm worried about the
> economic
> >incentives to help concentrate the collaborative effort required for this.
> >
> >Also, there will be difficulty getting people with existing systems to switch
> >over to something new. That's going to be a lot of pain for not much near-
> term
> >gain.  Most people just cobble together their own system, get to a fixed
> level of
> >automation, then move on to testing or board usage within that
> framework.
> >
> >To be a bit more blunt about it, as a concrete example: would LAVA adopt a
> new
> >system for low-level board control, if it presented itself?  I kind of doubt it.
> >Who would do this work?
> 
> That's the big question, yes. In the LAVA team, we've already solved a
> lot of the issues that we've seen, in our own ways. We want to be good
> Open Source citizens (of course!), but it's difficult to justify
> spending much engineering time on ripping things out and moving to
> different underpinnings unless we can see concrete benefit. I'd expect
> most teams to be the same. Chicken and egg, as you said earlier. The
> key thing to make it all worthwhile will be to show the value of doing
> it, while minimising the cost. Let's see where we can take that.

Steve - good to see your response.

I agree with the 'difficult to justify... without concrete benefit' idea.

I'm struggling with trying to decide what system to put my own efforts
into, and to use with Fuego.  My understanding of the different systems
is somewhat limited, but I'm aware of at least the following:
 - r4d - linutronix
 - ttc - Sony (well, mostly me and my lab)
 - labgrid - pengutronix
 - pduclient/LAVA - Linaro

One thing I've observed is that there are substantial differences
in the level of abstraction and the board management architecture
for these different systems.  I previously had hoped we could do a simple
survey of the different systems, and come up with a set of verbs
that everyone could use - making it possible for, say, tests written
in LAVA to run in a farm that used labgrid or r4d.  I would certainly like
to easily run Fuego tests in LAVA-based farms.  Actually, I'd like to make
Fuego independent of the board farm management software, so the
user can choose.

However, it has turned out to not be that simple.  Different systems
impose different hardware requirements, or make assumptions about
the nature of system deployment, that seem incompatible with
other systems.  Or, at least that's my understanding.

I'm not sure where to start, but maybe it would be good to have a
discussion about the basic requirements and operation of each
system, to see where the compatibilities and incompatibilities are.
(Or, maybe start with the requirements that the upper-level
software or user have, that are using the different systems).

Would it be worth putting together some kind of "board farm summit"
at the next plumbers?  I would suggest ELC, but it's already programmed,
and coming up soon, and Linaro has Connect going on the same month, so
I expect travel would be an issue.

Let me know what you think.
 -- Tim