[Automated-testing] Farming together - areas of collobration

[Bird, Timothy]

> -----Original Message-----
> From: Kieran Bingham on Saturday, November 11, 2017 7:37 AM
> Thanks for taking the time to get things started!
> 
> On 11/11/17 00:23, Andrew Murray wrote:
> > Hello,
> >
> > Following on from the "Farming Together" BOF at ELCE [1][2], I'm keen
> > to collaborate with this new community to harmonise automated testing
> > infrastructure tools.
> >
> > It's my view that there are two main use-cases for farms - the first
> > being to facilitate automated testing and the second to facilitate
> > remote working (interactive/real-time access to hardware). Any efforts
> > we undertake should have these use-cases in mind - as it seems that
> > much of the existing software available is narrowly targeted at a
> > specific task or user within one of these use-cases. (Are there any
> > other use cases?).
> 
> I agree, though for me - both of those are part of the same requirement.
> 
> That is to say - for me - I envisage that a developer should be able to 'lock' a
> board, take control of it, and use it and then release it. (Or a timeout would
> release the lock as well perhaps)

I agree with this concept.  'ttc' has board reservation.  (I'm not saying this
promote 'ttc', but rather to indicate that it was one of the things we added
based on our usage models for board farms at Sony.)  I'm not sure
where board reservation fits in the layer model - probably at the "cow"
layer.  (BTW - the layers will need non-analogy names at some point.)

> 
> An automated test system - should then simply be another 'developer' who
> just
> happens to be an automated build and test system. Whether a real person,
> or a
> build pc - the access should be much the same. (albeit perhaps with their
> own
> unique access keys and permissions of course)

I think this makes the model of managing the board for multiple use cases
simpler.  If there are differences between human/board interaction
and automated-system/board interaction, it would be good to identify
those.

> 
> 
> > My vision of a farm is that it would consist of three distinct layers...
> >
> > The Grass - This is the bottom layer which provides the bare minimum
> > software abstraction to the fundamental capabilities of the farm. This
> > consists of physical devices and suitable drivers for them. The
> > interface to higher layers is of the language of turn power port 2 on,
> > bridge relay 5 on, etc. The goal here is for us to be able pick up
> > some hardware off the shelf (e.g. a APC power switch) and to have a
> > 'driver' already available from this community that fits into this
> > view of the world. In order to achieve this it would be necessary to
> > define categories of devices (power, relay, serial), the verbs (power
> > cycle, power off, etc) and a way of addressing physical devices. I
> > believe the LabGrid stack has a good model in this respect.
> 
> I had started looking at this part - and then having come across labgrid - I
> believe I will start adapting this rather than rewrite a competing system.
> 
> My requirement here is abstracting the physical hardware out to a
> description
> (like YAML) that is easy to update when the hardware is moved. And I think
> labgrid covers this.
>  
> > The Cow - This is the middle layer which uses the bottom layer and
> > provides an abstraction of a board, this is where boards are defined
> > and related to the physical devices. This layer would manage exclusive
> > access to boards and self-tests. Users of this layer could find out
> > what boards are available (access control), what their capabilities
> > are and access to those capabilities. The language used here may be
> > turn board on, get serial port, press BOOT switch, etc. The goal here
> > is that we can work as a community to create definition of boards - a
> > bit like device tree at a high physical level.
> 
> I think this layer is quite intrinsically linked with the lower layer, so it's
> hard for me to visualise them as separate.
> 
> I guess in reality - it is this layer that I had already started looking at, and
> I had a vision to implement an access system over SSH similar to gitlab.
> 
> I started prototyping (though don't expect much here yet):
> 	https://github.com/kbingham/lab.git
> 
> 
> My design goals here are that an individual should be able to access targets
> with name based resolution such as:
> 
>   lab beaglebone serial # Load up a shared serial console over tmux/screen
>   lab beaglebone on
>   lab beaglebone reboot
>   lab beaglebone upload Image board.dtb   # Upload boot files using rsync/scp
> 

Just by way of comparison, ttc supports similar things:
Assuming a board named 'bbb' for "beagleboneblack".  I use commands like the following
on a daily basis:
 $ ttc bbb console - get serial console
 $ ttc bbb login - get network login (usually ssh these days)
 $ ttc bbb reboot - reboot board
 $ ttc bbb kinstall - install kernel
 $ ttc bbb fsinstall - install file system (or image, as appropriate)
 $ ttc run <command> - execute command on board

See https://elinux.org/Ttc_Program_Usage_Guide for a list of verbs.
(Again - I'm not promoting ttc, just providing data for comparison with other systems).
Having tried to push ttc in the past as an industry standard, and gotten nowhere,
I am perhaps more sensitive to the issues of how hard it is to build an ecosystem
around this stuff.

Maybe I should try to convert my lab over to labgrid, and see if it handles
the things I want to do (and if not send some patches upstream).
For some reason, I could never muster the energy to do this with LAVA,
but maybe converting to labgrid wouldn't involve as much perceived pain.

Is anyone on this list using labgrid (or especially if they've converted over
from something else), and can tell me how painful it was?

> 
> By using the authorized_keys file of SSH to redirect all connections through a
> parsing script we can have some fun in manipulating what the request
> actually
> does. It also provides authentication, and security. ( I hope SSH is still
> considered secure )
> 
> Anyway, I envisage this being backed by some database storage or such
> which will
> maintain connection states, and permissions, and maybe a web-front end
> too.

It would be good if things were discoverable, inspectable, etc.  That's
one thing that I never finished with ttc - was a web front-end.  My
question would be if there are existing front-ends that this could plug
in to (Jenkins, kernelCI, LAVA - do they all have "board management" screens?)

> 
> 
> > The Dairy - This is the top layer which uses the middle layer. This is
> > the application layer, applications can focus on their value-add which
> > may be running tests (LAVA) or providing remote access to boards
> > (perhaps with additional features such as 'workspaces' with prebuilt
> > and predeployed software images).
> 
> I like the idea of prebuilt workspaces or such ...
> 
> 
> > The problem we have at the moment is that existing software tries to
> > do all three. For example LAVA - it's a great 'Dairy' (application)
> > but the person setting up LAVA on a machine has to plumb it into their
> > physical hardware. This normally creates a easily breakable link that
> > overly couples the hardware to LAVA. It means LAVA has to do more than
> > just schedule and run tests - it has to manage boards, it has to check
> > they are OK, etc. This is a distraction for LAVA - wouldn't it be
> > great if we tell it where our 'Cow' is and it figures out the rest.
> >
> > By splitting the stack into layers, we can move beyond the basics of
> > farms and really improve the reliability of farms and add features. We
> > can also allow there to be multiple applications, e.g. a board farm
> > that is kernelci enabled but also allows remote access for developers.
> >
> > In my experience, the most difficult part of farming is making it
> > robust and reliable. It would be nice to not be scared of replacing a
> > USB hub without fear of everything breaking and boards seeing the
> > wrong UARTs etc. It would be nice to be alerted when a board is no
> > longer working, etc.
> >
> > I'm keen to focus on one small part of this stack and do it well - I
> > don't see the need to abandon existing projects in this field, but
> > instead to align in certain areas.
> >
> > In my view it's the bottom two layers that need the most work. I have
> > a particular vision for how part of the 'Grass' layer should work..
> 
> I think perhaps we should meet up for a beer to discuss this sometime!
> (farmers-con anyone?)

Until there's sufficient ecosystem to justify a full event, I can always
approve/secure a session or two at ELC or ELCE, for BOFs, talks or even working
meetings.

 -- Tim