[Automated-testing] Farming together - areas of collobration

[Kieran Bingham]

Hi Andy,

Thanks for taking the time to get things started!

On 11/11/17 00:23, Andrew Murray wrote:
> Hello,
> 
> Following on from the "Farming Together" BOF at ELCE [1][2], I'm keen
> to collaborate with this new community to harmonise automated testing
> infrastructure tools.
> 
> It's my view that there are two main use-cases for farms - the first
> being to facilitate automated testing and the second to facilitate
> remote working (interactive/real-time access to hardware). Any efforts
> we undertake should have these use-cases in mind - as it seems that
> much of the existing software available is narrowly targeted at a
> specific task or user within one of these use-cases. (Are there any
> other use cases?).

I agree, though for me - both of those are part of the same requirement.

That is to say - for me - I envisage that a developer should be able to 'lock' a
board, take control of it, and use it and then release it. (Or a timeout would
release the lock as well perhaps)

An automated test system - should then simply be another 'developer' who just
happens to be an automated build and test system. Whether a real person, or a
build pc - the access should be much the same. (albeit perhaps with their own
unique access keys and permissions of course)


> My vision of a farm is that it would consist of three distinct layers...
> 
> The Grass - This is the bottom layer which provides the bare minimum
> software abstraction to the fundamental capabilities of the farm. This
> consists of physical devices and suitable drivers for them. The
> interface to higher layers is of the language of turn power port 2 on,
> bridge relay 5 on, etc. The goal here is for us to be able pick up
> some hardware off the shelf (e.g. a APC power switch) and to have a
> 'driver' already available from this community that fits into this
> view of the world. In order to achieve this it would be necessary to
> define categories of devices (power, relay, serial), the verbs (power
> cycle, power off, etc) and a way of addressing physical devices. I
> believe the LabGrid stack has a good model in this respect.

I had started looking at this part - and then having come across labgrid - I
believe I will start adapting this rather than rewrite a competing system.

My requirement here is abstracting the physical hardware out to a description
(like YAML) that is easy to update when the hardware is moved. And I think
labgrid covers this.



> The Cow - This is the middle layer which uses the bottom layer and
> provides an abstraction of a board, this is where boards are defined
> and related to the physical devices. This layer would manage exclusive
> access to boards and self-tests. Users of this layer could find out
> what boards are available (access control), what their capabilities
> are and access to those capabilities. The language used here may be
> turn board on, get serial port, press BOOT switch, etc. The goal here
> is that we can work as a community to create definition of boards - a
> bit like device tree at a high physical level.


I think this layer is quite intrinsically linked with the lower layer, so it's
hard for me to visualise them as separate.

I guess in reality - it is this layer that I had already started looking at, and
I had a vision to implement an access system over SSH similar to gitlab.

I started prototyping (though don't expect much here yet):
	https://github.com/kbingham/lab.git


My design goals here are that an individual should be able to access targets
with name based resolution such as:

  lab beaglebone serial # Load up a shared serial console over tmux/screen
  lab beaglebone on
  lab beaglebone reboot
  lab beaglebone upload Image board.dtb   # Upload boot files using rsync/scp


By using the authorized_keys file of SSH to redirect all connections through a
parsing script we can have some fun in manipulating what the request actually
does. It also provides authentication, and security. ( I hope SSH is still
considered secure )

Anyway, I envisage this being backed by some database storage or such which will
maintain connection states, and permissions, and maybe a web-front end too.


> The Dairy - This is the top layer which uses the middle layer. This is
> the application layer, applications can focus on their value-add which
> may be running tests (LAVA) or providing remote access to boards
> (perhaps with additional features such as 'workspaces' with prebuilt
> and predeployed software images).

I like the idea of prebuilt workspaces or such ...


> The problem we have at the moment is that existing software tries to
> do all three. For example LAVA - it's a great 'Dairy' (application)
> but the person setting up LAVA on a machine has to plumb it into their
> physical hardware. This normally creates a easily breakable link that
> overly couples the hardware to LAVA. It means LAVA has to do more than
> just schedule and run tests - it has to manage boards, it has to check
> they are OK, etc. This is a distraction for LAVA - wouldn't it be
> great if we tell it where our 'Cow' is and it figures out the rest.
> 
> By splitting the stack into layers, we can move beyond the basics of
> farms and really improve the reliability of farms and add features. We
> can also allow there to be multiple applications, e.g. a board farm
> that is kernelci enabled but also allows remote access for developers.
> 
> In my experience, the most difficult part of farming is making it
> robust and reliable. It would be nice to not be scared of replacing a
> USB hub without fear of everything breaking and boards seeing the
> wrong UARTs etc. It would be nice to be alerted when a board is no
> longer working, etc.
> 
> I'm keen to focus on one small part of this stack and do it well - I
> don't see the need to abandon existing projects in this field, but
> instead to align in certain areas.
> 
> In my view it's the bottom two layers that need the most work. I have
> a particular vision for how part of the 'Grass' layer should work..

I think perhaps we should meet up for a beer to discuss this sometime!
(farmers-con anyone?)


> I find that when managing a farm, it would be nice to add new boards
> without having to change some integral farm configuration or have
> particular access rights to 'the farm' to plumb in a new board. I.e.
> so that anyone can add a farm without risk of breaking anything else.
> In our farm we install boards on rack-mount shelves - I think it would
> be a fantastic idea if we could define a standard that:
> 
> - Standardises connectivity between a shelf and the farm PC (perhaps
> just USB, Ethernet and power)
> - Requires each shelf to contain a USB stick with a file that
> describes the board, the physical USB topology between the shelf and
> its components (USB serial adaptors etc)
> 

Intriguing - this is an extra abstraction layer that I had not thought of (the
'shelf')

> This immediately opens up some benefits:
> 
> - You can now make a new piece of hardware available to farm users by
> creating a shelf and plugging it into the farm - no need to go and
> configure the farm - this makes it really easy.
> - Likewise you can plug your shelf into any shelf of the farm - i.e.
> makes maintenance really easy
> - You could plug your shelf into your PC and farm software on your PC
> would detect the hardware and give you access.
> - You could plug your shelf into anyone elses farm.

Ok - well this is just 'moving' the configuration onto a USB stick... which will
still need updating as the shelf evolves...


> As a slightly more concrete example, imagine you have a beagle bone
> black, there would be a community definition file available that
> describes that it is a board named 'Beagle Bone Black' which describes
> that it has a power jack, a power button, a reset button, serial, USB
> and an Ethernet cable. Perhaps the definition file has additional
> information for example that to power on the board you have to apply
> power and then press the power button. You could then create a shelf
> with the power connected, a USB relay for the power button and serial.
> You could then create a mapping file on a USB stick that describes:

Isn't the information that a board has a power connection implicit?
For me the information on *how* to power that board is more important:

# power.yaml
power:
  acme1:
    driver: baylibre-acme	# provides 'on/off/reboot/status'
    ports: 8

  energenie:
    driver: sispmctl
    device: 0
    host: raspberrypi.local 	# Objects on a different 'server'

# Boards.yaml (or support a file per 'shelf'?)
boards:
  beaglebone:
    serial: /dev/serial/by-path/platform-3f980000.usb-usb-0:1.4:1.0-port0
    power:
     - acme1 port-3      # Not sure how to 'cross reference' in yaml yet ...
     - energenie port-4  # I have a scenario that needs two switches enabled


I assume (I need to check still) that the above is something like what labgrid
is providing.


> - That this relates to a community definition file called BBB.
> - BBB Power is connected to the shelf power supply
> - BBB power button is connected to a USB relay of type XXX on the
> shelf with a specific relative physical path (from the end of the
> shelf) using port 1
> - BBB serial is connected to a USB serial on the shelf with a specific
> relative physical path (from the end of the shelf)
> - The remainder of the other connections are not made.
> 
> When the farm detects the USB it can present a BBB to the user with
> information about what capabilities are available. The user can then
> interact using BBB terminology - e.g. press BBB power button, or more
> generally - e.g. turn board on.
> 
> Of course I've made many assumptions here about shelves, the hardware
> within it etc - but this is just an idea for discussion.
> 
> Would this split of layers be beneficial?
> 
> Would such a standard be helpful?
> 
> What areas of farming need to be addressed?

Boot methods? Abstracting data storage?
 NFS Root, TFTP server for Kernel Images
 Boards that boot from SD card (I see a lot of people are looking at SDMux'es)
 USB DFU / Android / 96boards style boot?


> Is this the right place to start?

Of course - we need to start somewhere!


> Keen to all feedback. (Apologies for the rambling and terrible farm
> analogies :D)


Well - I think cows are a bit big. I was thinking of farming things like
chickens, but it seems the reality is that it's more like rabbits based on the
speed at which the boards seem to breed and multiply when you start a farm. ;-)


> [1] - BoF Slides - https://www.elinux.org/images/5/52/BoF-FarmingTogether.pdf
> [2] - Farming Together Wiki - https://www.elinux.org/Board_Farm
> 
> Thanks,
> 
> Andrew Murray
> 

Cheers

--
Kieran