[Automated-testing] automated testing summit

[Tim.Bird at sony.com]

Heiko,

Please see comments inline below.

> -----Original Message-----
> From: Heiko Schocher on Thursday, October 25, 2018 7:34 PM
> 
> Hello all,
> 
> it was great to see you all on the ATS and we all saw, that
> automated testing is not a "one day" task ...
> 
> Here just my 2 cents of my tbot view of things, may it gives
> an idea, how we can structure "testing" ...
> 
> "tbot" view of the tasks "power on/off" and connect to a DUT
> console:
> 
> In tbots point of view "all is a testcase". You start tbot
> with a lab and a DUT configuration file. So tbot already
> "knows" to test which DUT in which lab. This decision must
> be met before calling tbot. This is easy, when you use tbot
> for your daily work, but maybe more complex, if you setup
> a CI ... but solveable.
> 
> May needed for understanding:
> In tbot you can open as much connections you want to have,
> and write testcases, which work on this connections. So
> you can call a testcase which set a gpio pin on the connection
> which has the DUT console, but also on a connection which
> control your DUT in the lab.
> 
> If tbot or a testcase decides to "power on/off" the DUT, it
> is simply calling a testcase!

tbot has an interesting way of composing tests.  But the problem
I have with this design is that it does not actually address the testing
activities itself, but rather appears to leave all definition of test operations
to the user.   Saying that someone can define something with your
framework, is not the same thing as having your framework actually
support that.  This requires more design and implementation than
merely leaving it as an exercise for the user.

APIs need to be defined, and protocols implemented, in order for
different test systems to interoperate.  I don't see how the design
of tbot  accomplishes this.

> 
> Therefore in tbot you have (not explicitely marked!) a
> "generic testcase" which powers on/off DUT with name Y.
> 
> This generic (or more higher level?) testcase now only
> looks into the configuration variable, which contains
> the name of the testcase which it has to use for
> power on the DUT and calls it. If interested in details
> see [1].
> 
> This could be now in the easiest case simply call testcase
> "set GPIO pin number X to state Y" [2] (Remark this "set GPIO
> pin" testcase may also needs to do some setup tasks, so not
> always only one command)
> 
> An example for using another testcase for control power
> you find in [3]
> 
> A more complex testcase for this task is for example:
> 
> You have connected a debugger to the DUT, which set
> the DUT in halt after powering on... so you have to go
> into the debuggers commandline and send the "go" command,
> (yes, of course this is also a testcase from tbots sight)
> than call a testcase which checks on the console if you get
> a bootloader banner and bootloader commandshell.
> If you get not the banner after n seconds, call testcase,
> which may unbreak your DUT with the help of the debugger,
> call again testcase, which checks if you get bootloader
> commandshell...
> 
> Uff, yeah, things can get complicated, but from the generic
> sight, it is "only" power on the DUT ... and as you can call
> testcases from a testcase (in tbot), you can write "generic"
> testcases, which call for DUT or lab specific tasks, specific
> testcases. If there is no tescase for your special setup in
> your lab, yeah, you must write a new testcase (or use in parts
> where possible already existing testcases).
> 
> The same fits for "getting the console". In tbot there is
> the config variable "tc_lab_denx_connect_to_board_tc"
> (I really should give them generic names...)
> Which you can set in the lab config file (it makes no sense
> in the DUT config file, as this should be not DUT specific)
> BTW: you can set config variables from tbots commandline,
> so you could change which testcase tbot uses for connecting
> to the console also from there...
> 
> And now how you connect to the DUTs console? Correct, its a
> testcase so you can use kermit, ssh, jtag,.... whatever.
> If there is no testcase for it, write a testcase, push it to
> mainline, and everbody can use it ...

For any framework feature, the question is not whether
someone can write a facility to accomplish something, but
whether the code for the feature is already written
and available in a format that makes it possible to share and re-use
by other users and developers.  How would Fuego or KernelCI
or YoctoProject or LAVA or 0-day use a tbot jtag testcase
to add that functionality to their lab if they wanted to?

How many console-access mechanisms does tbot already
include in its repository?  Is the number sufficient to make
these other projects invest the work to support a new API
to use tbot modules?

How does the tbot "API" and ecosystem make it possible
for even tbot users to share their modules with each other,
other than checking them into the same git repository?
For example, how do the authors of separate modules agree on
the names, types and values of variables that must be passed
from one layer of the test stack to another?
> 
> Another "generic" approach for "git biesct" you find here:
> https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py
> (just saw there are U-Boot comments ... ignore it, it is
>   not longer u-boot specific)
> 
> I hope you get the idea ...
> 
> Maybe this is a generic approach for defining tests?
There's more to defining the test stack than the composition
of operations.  There are issues with where processes run,
whether they are distributed or centralized, and what language
bindings are supported for APIs (python is popular, but many
of the existing systems use other base languages like go or ruby).
There are issues with synchronous vs. asynchronous operation,
whether and where data should persist, and how it should be factored.

It is not feasible to support a complex test system by just composing
smaller elements together on a single host.

If there are specific design elements of tbot that are usable for portions
of the test stack, I think we should look at them for consideration.

For example, your example below uses the API set_power_state()
to control power to a board.  I'm not sure if that's the full API or if there
are other calls or arguments required for a generic sub-system.
I have yet to look at the pdudaemon code to see what its power
control API looks like.  Comparing and contrasting
the two APIs might be useful, to see if there is some functionality
that either system has that could be re-used in the other.

Regards,
 -- Tim


> 
> define "generic" testcases, which call for tasks, which are
> not generic, testcases, which can be configured DUT or lab
> specific.
> 
> bye,
> Heiko
> [1] calling testcase for power on/off
> https://github.com/hsdenx/tbot/blob/master/src/tc/tc_lab_poweroff.py
> which calls:
> https://github.com/hsdenx/tbot/blob/master/src/common/tbotlib.py#L446
> which calls testcase with name
> self.config.tc_lab_denx_power_tc
> 
> I must admit, that I have a part of this task in the tbotlib,
> which is not necessary anymore, I think, and of course the
> name of the variable is a little bit denx centric sight of
> view ... default value is use denx specific testcase ...
> 
> [2] set gpio pin
> https://github.com/hsdenx/tbot/blob/master/src/tc/linux/tc_workfd_set_g
> pio.py
> 
> [3] set power on/off testcase for a gembird powercontroller
> https://github.com/hsdenx/tbot/blob/master/config/lab_home.py#L15