[Automated-testing] Farming together - areas of collobration

[Andrew Murray]

Hello,

Following on from the "Farming Together" BOF at ELCE [1][2], I'm keen
to collaborate with this new community to harmonise automated testing
infrastructure tools.

It's my view that there are two main use-cases for farms - the first
being to facilitate automated testing and the second to facilitate
remote working (interactive/real-time access to hardware). Any efforts
we undertake should have these use-cases in mind - as it seems that
much of the existing software available is narrowly targeted at a
specific task or user within one of these use-cases. (Are there any
other use cases?).

My vision of a farm is that it would consist of three distinct layers...

The Grass - This is the bottom layer which provides the bare minimum
software abstraction to the fundamental capabilities of the farm. This
consists of physical devices and suitable drivers for them. The
interface to higher layers is of the language of turn power port 2 on,
bridge relay 5 on, etc. The goal here is for us to be able pick up
some hardware off the shelf (e.g. a APC power switch) and to have a
'driver' already available from this community that fits into this
view of the world. In order to achieve this it would be necessary to
define categories of devices (power, relay, serial), the verbs (power
cycle, power off, etc) and a way of addressing physical devices. I
believe the LabGrid stack has a good model in this respect.

The Cow - This is the middle layer which uses the bottom layer and
provides an abstraction of a board, this is where boards are defined
and related to the physical devices. This layer would manage exclusive
access to boards and self-tests. Users of this layer could find out
what boards are available (access control), what their capabilities
are and access to those capabilities. The language used here may be
turn board on, get serial port, press BOOT switch, etc. The goal here
is that we can work as a community to create definition of boards - a
bit like device tree at a high physical level.

The Dairy - This is the top layer which uses the middle layer. This is
the application layer, applications can focus on their value-add which
may be running tests (LAVA) or providing remote access to boards
(perhaps with additional features such as 'workspaces' with prebuilt
and predeployed software images).

The problem we have at the moment is that existing software tries to
do all three. For example LAVA - it's a great 'Dairy' (application)
but the person setting up LAVA on a machine has to plumb it into their
physical hardware. This normally creates a easily breakable link that
overly couples the hardware to LAVA. It means LAVA has to do more than
just schedule and run tests - it has to manage boards, it has to check
they are OK, etc. This is a distraction for LAVA - wouldn't it be
great if we tell it where our 'Cow' is and it figures out the rest.

By splitting the stack into layers, we can move beyond the basics of
farms and really improve the reliability of farms and add features. We
can also allow there to be multiple applications, e.g. a board farm
that is kernelci enabled but also allows remote access for developers.

In my experience, the most difficult part of farming is making it
robust and reliable. It would be nice to not be scared of replacing a
USB hub without fear of everything breaking and boards seeing the
wrong UARTs etc. It would be nice to be alerted when a board is no
longer working, etc.

I'm keen to focus on one small part of this stack and do it well - I
don't see the need to abandon existing projects in this field, but
instead to align in certain areas.

In my view it's the bottom two layers that need the most work. I have
a particular vision for how part of the 'Grass' layer should work...

I find that when managing a farm, it would be nice to add new boards
without having to change some integral farm configuration or have
particular access rights to 'the farm' to plumb in a new board. I.e.
so that anyone can add a farm without risk of breaking anything else.
In our farm we install boards on rack-mount shelves - I think it would
be a fantastic idea if we could define a standard that:

- Standardises connectivity between a shelf and the farm PC (perhaps
just USB, Ethernet and power)
- Requires each shelf to contain a USB stick with a file that
describes the board, the physical USB topology between the shelf and
its components (USB serial adaptors etc)

This immediately opens up some benefits:

- You can now make a new piece of hardware available to farm users by
creating a shelf and plugging it into the farm - no need to go and
configure the farm - this makes it really easy.
- Likewise you can plug your shelf into any shelf of the farm - i.e.
makes maintenance really easy
- You could plug your shelf into your PC and farm software on your PC
would detect the hardware and give you access.
- You could plug your shelf into anyone elses farm.

As a slightly more concrete example, imagine you have a beagle bone
black, there would be a community definition file available that
describes that it is a board named 'Beagle Bone Black' which describes
that it has a power jack, a power button, a reset button, serial, USB
and an Ethernet cable. Perhaps the definition file has additional
information for example that to power on the board you have to apply
power and then press the power button. You could then create a shelf
with the power connected, a USB relay for the power button and serial.
You could then create a mapping file on a USB stick that describes:

- That this relates to a community definition file called BBB.
- BBB Power is connected to the shelf power supply
- BBB power button is connected to a USB relay of type XXX on the
shelf with a specific relative physical path (from the end of the
shelf) using port 1
- BBB serial is connected to a USB serial on the shelf with a specific
relative physical path (from the end of the shelf)
- The remainder of the other connections are not made.

When the farm detects the USB it can present a BBB to the user with
information about what capabilities are available. The user can then
interact using BBB terminology - e.g. press BBB power button, or more
generally - e.g. turn board on.

Of course I've made many assumptions here about shelves, the hardware
within it etc - but this is just an idea for discussion.

Would this split of layers be beneficial?

Would such a standard be helpful?

What areas of farming need to be addressed?

Is this the right place to start?

Keen to all feedback. (Apologies for the rambling and terrible farm
analogies :D)

[1] - BoF Slides - https://www.elinux.org/images/5/52/BoF-FarmingTogether.pdf
[2] - Farming Together Wiki - https://www.elinux.org/Board_Farm

Thanks,

Andrew Murray