[Automated-testing] Board management API discussion at ATS - my ideas

[Tim.Bird at sony.com]

Hello everyone,

I have a few ideas about board management APIs that I thought I'd share.  There will
be a discussion about these at ATS, but I thought I'd share some of my ideas ahead of
time to see if we can get some discussion out of the way before the event - since time
at the event will be somewhat limited.

What I'd like to see in a "standard" board management API is a system whereby 
any test framework can be installed in a lab, and
1) automatically detect the board management layer that is being used in the lab
2) be able to use a single set of APIs (functions or command line verbs) to 
communication with the board management layer
3) possibly, a way to find out what features are supported by the board management
layer (that is, introspection)

The following might be nice, but I'm not sure:
4) the ability to support more than one board management layer in a single lab

= style of API =
My preference would be to have the IPC from the test manager (or test scheduler) to
the board management layer be available as a Linux command line.  I'm OK with having a
python library, as that's Fuego's native language, but I think most board management systems
already have a command line, and that's more universally accessible by test frameworks.
Also, it should be relatively easy to create a command line interface for libraries that
currently don't have one (ie only have a binding in a particular language (python, perl, C library, etc.))

I don't think that the operations for the board management layer are extremely time-sensitive,
so I believe that the overhead of going through a Linux process invocation to open a separate
tool (especially if the tool is in cache) is not a big problem.  In my own testing, the overhead of invoking
the 'ttc' command line (which is written in python) takes less than 30 milliseconds, when python
and ttc are in the Linux cache.  I think this is much less than the time for the operation that are
actually performed by the board management layer.

As a note, avoiding a C or go library (that is a compiled language) avoids having to re-compile
the test manager to  communicate with different board management layers. 

For detection, I propose something like placing a file into a well-known place in a Linux filesystem,
when the board management layer is installed.

For example, maybe making a script available at:
/usr/lib/test/test.d
(and having scripts: lava-board-control, ttc-board-control, labgrid-board-control, beaker-board-control,
libvirt-board-control, r4d-board-control, etc)

or another alternative is to place a config script for the board management system in:
/etc/test.d
with each file containing the name of the command line used to communicate with that board management layer, and
possibly some other data that is required for interface to the layer (e.g. the communication method, if we decide to
support more than just CLI (e.g. port of a local daemon, or network address for the server providing board management),
or location of that board management layer's config file.

= starting functions =
Here are some functions that I think  the board management layer should support:

introspection of the board management layer supported features:
verb: list-features

introspection of the board layer managed objects:
vefb: list-boards

reserving a board:
verbs: reserve and release

booting the board:
verb: reboot
(are power-on and power-off needed at this layer?)

operating on a board:
   get serial port for the board
   verb: get-serial-device
(are higher-level services needed here, like give me a file descriptor for a serial connection to the device?  I haven't used
terminal concentrators, so I don't know if it's possible to just get a Linux serial device, or maybe a Linux pipe name, and
have this work)

  execute a command on the board and transfer files
  verbs: run, copy_to, copy_from

Now, here are some functions which I'm not sure belong at this layer or another layer:
  provision board:
  verbs: install-kernel,  install-root-filesystem
  boot to firmware?

Here are some open questions:
 * are all these operations synchronous, or do we need some verbs that do 'start-an-operation', and 'check-for-completion'?
    * should asynchronicity be in the board management layer, or the calling layer? (if the calling layer, does the board
    management layer need to support running the command line in concurrent instances?)
 * are these sufficient for most test management/test scheduler layers? (ie are these the right verbs?)
 * what are the arguments or options that go along with these verbs?
    * e.g. which ones need timeouts? or is setting a timeout for all operations a separate operation itself?
 * for provisioning verbs:
   * how to express where the build artifacts are located (kernel image, rootfs) to be used for the operations?
      * just local file paths, or an URL for download from a build artifact server?
  * do we need to consider security as part of initial API design?  (what about user ID, access tokens, etc.)

I've started collecting data about different test management layers at:
https://elinux.org/Board_Management_Layer_Notes

Let me know what you think.
 -- Tim0