[Automated-testing] automated testing summit

[Heiko Schocher]

Hello all,

it was great to see you all on the ATS and we all saw, that
automated testing is not a "one day" task ...

Here just my 2 cents of my tbot view of things, may it gives
an idea, how we can structure "testing" ...

"tbot" view of the tasks "power on/off" and connect to a DUT
console:

In tbots point of view "all is a testcase". You start tbot
with a lab and a DUT configuration file. So tbot already
"knows" to test which DUT in which lab. This decision must
be met before calling tbot. This is easy, when you use tbot
for your daily work, but maybe more complex, if you setup
a CI ... but solveable.

May needed for understanding:
In tbot you can open as much connections you want to have,
and write testcases, which work on this connections. So
you can call a testcase which set a gpio pin on the connection
which has the DUT console, but also on a connection which
control your DUT in the lab.

If tbot or a testcase decides to "power on/off" the DUT, it
is simply calling a testcase!

Therefore in tbot you have (not explicitely marked!) a
"generic testcase" which powers on/off DUT with name Y.

This generic (or more higher level?) testcase now only
looks into the configuration variable, which contains
the name of the testcase which it has to use for
power on the DUT and calls it. If interested in details
see [1].

This could be now in the easiest case simply call testcase
"set GPIO pin number X to state Y" [2] (Remark this "set GPIO
pin" testcase may also needs to do some setup tasks, so not
always only one command)

An example for using another testcase for control power
you find in [3]

A more complex testcase for this task is for example:

You have connected a debugger to the DUT, which set
the DUT in halt after powering on... so you have to go
into the debuggers commandline and send the "go" command,
(yes, of course this is also a testcase from tbots sight)
than call a testcase which checks on the console if you get
a bootloader banner and bootloader commandshell.
If you get not the banner after n seconds, call testcase,
which may unbreak your DUT with the help of the debugger,
call again testcase, which checks if you get bootloader
commandshell...

Uff, yeah, things can get complicated, but from the generic
sight, it is "only" power on the DUT ... and as you can call
testcases from a testcase (in tbot), you can write "generic"
testcases, which call for DUT or lab specific tasks, specific
testcases. If there is no tescase for your special setup in
your lab, yeah, you must write a new testcase (or use in parts
where possible already existing testcases).

The same fits for "getting the console". In tbot there is
the config variable "tc_lab_denx_connect_to_board_tc"
(I really should give them generic names...)
Which you can set in the lab config file (it makes no sense
in the DUT config file, as this should be not DUT specific)
BTW: you can set config variables from tbots commandline,
so you could change which testcase tbot uses for connecting
to the console also from there...

And now how you connect to the DUTs console? Correct, its a
testcase so you can use kermit, ssh, jtag,.... whatever.
If there is no testcase for it, write a testcase, push it to
mainline, and everbody can use it ...

Another "generic" approach for "git biesct" you find here:
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py
(just saw there are U-Boot comments ... ignore it, it is
  not longer u-boot specific)

I hope you get the idea ...

Maybe this is a generic approach for defining tests?

define "generic" testcases, which call for tasks, which are
not generic, testcases, which can be configured DUT or lab
specific.

bye,
Heiko
[1] calling testcase for power on/off
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_lab_poweroff.py
which calls:
https://github.com/hsdenx/tbot/blob/master/src/common/tbotlib.py#L446
which calls testcase with name
self.config.tc_lab_denx_power_tc

I must admit, that I have a part of this task in the tbotlib,
which is not necessary anymore, I think, and of course the
name of the variable is a little bit denx centric sight of
view ... default value is use denx specific testcase ...

[2] set gpio pin
https://github.com/hsdenx/tbot/blob/master/src/tc/linux/tc_workfd_set_gpio.py

[3] set power on/off testcase for a gembird powercontroller
https://github.com/hsdenx/tbot/blob/master/config/lab_home.py#L15

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs at denx.de