[Automated-testing] Automated Testing Summit - Test Stack survey

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Punnaiah Choudary Kalluri 
> Hi Tim,
> 

Thanks for filling out the survey.  A few comments and questions are inline below.

> > -----Original Message-----
> > From: Tim.Bird at sony.com [mailto:Tim.Bird at sony.com]
> >
> > == Survey Questions ==
> > * What is the name of your test framework?
> 
> "regression_xlnx", it's a command line based tool. Our In house  CI
> framework will utilize this tool for building the software And executing the
> test cases.

OK - you'all should come up with a snappy name if you want any branding. :-)
I called this "Xilinux test", and put your survey response at:
https://elinux.org/Xilinx_test_survey_response


Is this test system available as open source?
Is it, or parts of it, published anywhere?

> 
> > Which of the aspects below of the CI loop does your test framework
> > perform?
> Build/deploy/test/analyze/publish the results
> 
> 
...
> > ** select an appropriate individual DUT based on SUT or test attributes?
> Yes


How does your system select the DUT?  For example, are tests and DUTs matched
by some tag, or capability/feature matching system?  How are the features
represented in the system (in text files, json, interactive query at run time, etc.)?

> 
> > ** reserve the DUT?
> Yes
> 
> > ** release the DUT?
> Yes
> 
> > * install the software under test to the DUT?
> Yes
> 
> > * install required packages before a test is run?
> Yes
> 
> > * require particular bootloader on the DUT? (e.g. grub, uboot, etc.)
> Yes.  but there is no requirement that after powerup bootloader is shown
> 
> > * deploy the test program to the DUT?
> Yes
> 
> > * prepare the test environment on the DUT?
> Yes
> 
> > * start a monitor (another process to collect data) on the DUT?
> No
> 
> > * start a monitor on external equipment?
> Yes
> 
> > * initiate the test on the DUT?
> Yes
> 
> > * clean up the test environment on the DUT?
> Yes
> 
> >
> > ==== DUT control ====
> > Does your test system:
> > * store board configuration data?
> 
> Yes
> 
> > ** in what format?
> Shell scripts
> 
> > * store external equipment configuration data?
> Yes
> 
> > ** in what format?
> Shells scripts
> 
> > * power cycle the DUT?
> Yes
> 
> > * monitor the power usage during a run?
> Yes
> 
> > * gather a kernel trace during a run?
> Yes
> 
> > * claim other hardware resources or machines (other than the DUT) for
> > use during a test?
> Yes

How is this done?  Is there an API in the test framework for claiming/releasing another
machine (besides the DUT)?

> 
> > * reserve a board for interactive use (ie remove it from automated
> testing)?
> Test framework is just another user of boardfarm managing SW which is
> running commands automatically. Interactive use is supported by default

Does the board farm managing software have another name (different from regression_xlnx)?
Is it a separate command line?

> 
> > * provide a web-based control interface for the lab?
> No
> 
> > * provide a CLI control interface for the lab?
> Yes
> 
> >
> > ==== Run artifact handling ====
> > Does your test system:
> > * store run artifacts
> > ** in what format?
> Txt files
> 
> > * put the run meta-data in a database?
> > ** if so, which database?
> No database. Its just a test case folder.
> 
> > * parse the test logs for results?
> Yes
> 
> > * convert data from test logs into a unified format?
> Yes
> 
> > ** if so, what is the format?
> Yaml, csv, html
> 
> > * evaluate pass criteria for a test (e.g. ignored results, counts or
> thresholds)?
> Yes
> 
> > * do you have a common set of result names: (e.g. pass, fail, skip,
> > etc.)
> Yes
> 
> > ** if so, what are they?
> PASS, FAIL, UNANALYZED
> 
> >
> > * How is run data collected from the DUT?
> Results will not be stored in DUT. It will be in the results folder
How is the data collected?  Just over SSH, or via the serial port?
You say the results are not stored in the DUT.  Does this mean collection
is during test execution over some network or serial transport?

(This is what LAVA does, but Fuego saves logs on the DUT, then moves
them to the testing host when the test is done.  I'm just trying to clarify
what your system does.)

I assume the results folder is on the testing host.

> 
> > ** e.g. by pushing from the DUT, or pulling from a server?
> > * How is run data collected from external equipment?
> mostly using the ssh
> 
> > * Is external equipment data parsed?
> Yes
> 
> 
> >
> > ==== User interface ====
> > Does your test system:
> > * have a visualization system?
> No
> 
> > * show build artifacts to users?
> Yes
> 
> > * show run artifacts to users?
> Yes
> 
> > * do you have a common set of result colors?
> > ** if so, what are they?
> yes. pass-green, fail-red, unanalyzed-yellow
> 
> > * generate reports for test runs?
> Yes
> 
> > * notify users of test results by e-mail?
> Yes
> 
> >> * can you query (aggregate and filter) the build meta-data?
> No
> 
> > * can you query (aggregate and filter) the run meta-data?
> No
> 
> >
> > * what language or data format is used for online results presentation?
> (e.g.
> > HTML, Javascript, xml, etc.)
> 'Html, java script
> 
> > * what language or data format is used for reports? (e.g. PDF, excel,
> > etc.)
> >
> web interface
> 
> > * does your test system have a CLI control tool?
> > ** what is it called?
> >
> It is not part of the test framework but we have custom CLI tool which call
> the test framework for build and run
> 
> > ==== Languages: ====
> > Examples: json, python, yaml, C, javascript, etc.
> > * what is the base language of your test framework core?
> Shell scripts, python and yaml
> 
> 
> >
> > What languages or data formats is the user required to learn?
> > (as opposed to those used internally)
> shell scripting and pyhon
> 
> 
> >
> > ==== Can a user do the following with your test framework: ====
> > * manually request that a test be executed (independent of a CI trigger)?
> Yes
> 
> > * see the results of recent tests?
> Yes
> 
> > * set the pass criteria for a test?
> Yes
> 
> > ** set the threshold value for a benchmark test?
> Yes

Is this part of the test definition?  How is it expressed?
Can you provide an example?


Thanks very much.
 -- Tim