[Automated-testing] Linaro Connect Report

Wed Apr 10 14:08:41 PDT 2019

Linaro Connect[0] was held last week in Bangkok. There were a lot of
sessions (listed below) around Linux testing, LAVA, and Fuego.

We also held an impromptu micro-summit (minutes below), where we all
locked ourselves in a room together for the day and discussed LAVA,
Fuego, test definitions, and yes, pdudaemon.

Finally, Maria Högberg (cc'd) has graciously arranged for a monthly
meeting that we can use to coordinate across projects throughout the
year - please send her an email if you'd like to be added. The first one
is scheduled for Thursday, May 9th at 13:00 UTC.

Selected Sessions (videos should be available within a week or two at the
referenced URLs):
[1] EAS Unit Testing
[2] LAVA Users Forum
[3] KEYNOTE: Open Source QA - what will it take to get to the next level
[4] Experiences and lessons we learned using kselftest and potential improvements.
[5] LAVA community enabled testing
[6] Harmonizing open source test definitions
[7] KernelCI New Generation
[8] What is this Fuego thing and where is it going?
[9] Automating test results analysis using neural networks
[10] Scheduling in CI/CD systems
[11] Bootloader testing in LAVA
[12] How to integrate Fuego automated testing tool in your CI loop

Micro-Summit Minutes

Date: Wednesday, April 4, 2019

Attendees
- Daniel Sangorrin (Toshiba)
- Tim Bird (Sony)
- Carlos Hernandez (TI)
- Kat Cosgrove (jfrog)
- Naresh Kamboju
- Antonio Tercerio
- Daniel Diaz
- Charles Oliveira
- Chase Qi
- Milosz Wasilewski
- Stevan Radakovic
- Dave Pigott
- Maria Högbergadding
- Luca Di Stefano
- Steve McIntyre
- Matt Hart
- Remi Duraffort
- Fathi Boudra
- Anders Roxell
- Dan Rue

PDU API
- Wider community agreement on standardizing on pdudaemon
- Linaro lab doesn’t actually use pdudaemon
- Running pdudaemon @ Linaro
  - [Dave] Queueing vs running immediately
    - [matt] we can just add it to pdu daemon if this is a problem
    - [Matt] added in a PR
- Dave will write a proposal for power control API
  - There will be an abstraction layer for power control to turn a device on or
    off.
- [Dave] What about pressing buttons, turning on/off usb ports, etc
  - [Tim] let’s not deal with composition right now, and only components
  - [Carlos] we do need to solve the problem… once we have an interface for
    power, we need a similar interface for relays.
  - Combinations of things may still make it complicated. For example, hold a
    button while pressing power.
- Core pdu api actions: on, off, status
- Core relay api actions: on, off, ???

USB control
- LAVA lab uses cambrionix USB hubs
- Also use good usb shielded cables
- Sony uses an open hardware board per DUT

Fuego Architecture
- Tim gave a subset of his Thursday talk; see his Thursday talk for the full
  details.
- Jenkins based
- Test execution system
  - Steps broken into discrete phases like build, deploy, run, process, etc.
  - Board and platform management are abstracted
- Not serial console based; typically uses SSH on a board that’s already
  provisioned (imaged)
- Contains functional and benchmark tests
- Fuego is a linux distribution, distributed as a container, self contained
- Testing driven from host, using ssh connection to DUT
- Is container privileged?
  - [tim] yes
  - [dave] Dynamic device detection is possible without privileged mode
- Designed for embedded linux testing
  - IoT possible
- By default, builds tests, though LTP has a way of checking to see if it’s on
  disk and skipping the build
  - Can also just skip the build phase so long as the build is in the location
    that is expected on target
- [long discussion about skip lists, known issues, and (lack of) test
  documentation]
  - Fuego-core has skip logic inside the test folder (ltp for example)
  - Test-definitions has similar
  - Additionally, lkft uses known issues in SQUAD to annotate known failures

LAVA Architecture
- Remi gave a brief LAVA architecture overview
- Components:
  - Lava-server
    - Apache2, lava-server-gunicorn, postgresql, lava-logs, lava-master
  - Lava-dispatcher
    - Lava-slave, lava-run, devices under test
- One lava-server, multiple dispatchers, multiple DUTs per dispatcher
- Lava-run is ephemeral process that communicates with DUT during a job run
- Device-type defines e.g. a raspberry pi
- Device defines an instance of a raspberry pi, with specifics about which port
  its plugged into, what serial port is it, etc.
- [daniel s] can i skip the deploy?
  - Yes, it’s supported
- [daniel s] parsing in the dispatcher, where is that done?
  - [steve] we recommend parsing gets done on DUT
  - [remi] interestedin junit, tap13 parsing in LAVA so that a parser on DUT
    isn’t necessary
  - [remi] goal is to not have to parse on DUT and do as much as we can on
    dispatcher
- Multinode jobs, used for networking, controlling lab hardware using lxc, etc
- Remi mentioned a feature to run a container as a part of a test run, which
  could help with jobs that are more complex, and eliminate a lot of existing
  multi-node jobs (making it easier)

Running Fuego tests in Lava
- Fuego operates a lot like an "interactive" LAVA test
- Are several solutions here:  One that seems good is to use the new feature to
  run a container as part of a test.

Running Linaro tests in Fuego
- Fuego already has Functional.linaro, which runs a Linaro test using
  testrunner.

Test Definitions
Tim Bird proposing `run-test.sh` as a naming convention for the name of the
test program for each package
- In ptest, you can build a test package, which you can install and execute.
- More controversially, why don’t you rename all the <testname>.sh scripts to
  run-test.sh
Tim Bird proposing a standard variable name for the location of kernel config
path
- LTP implements ‘KCONFIG_PATH’
How do we decide such standards?
- Proposal @ Automated testing list
- Tap13 is just a website. Maybe we need something similar to publish these
  things on
Tim Bird on Test Names
- Standardizing on test names would be nice so that we can compare across
  projects
- Related to test case definition discussion
- A good first step is to investigate how tests are named by various projects,
  starting with LTP.
  - How different are test case names for LTP?

Carlos Hernandez - Test case definition standards
Provide everything needed to run any test from any test framework.
- TGUID
- Test Name
- Description
- Test Execution Engine (e.g. LAVA, Fuego, VATF)
- Test script path

Next Steps
Agreement to start a public monthly meeting. Maria will organize.

[0] https://connect.linaro.org/
[1] https://bkk19.sched.com/event/L2DP/bkk19-114-eas-unit-testing
[2] https://bkk19.sched.com/event/L2EE/bkk19-120-lava-users-forum
[3] https://bkk19.sched.com/event/L2H8/bkk19-200k2-keynote-open-source-qa-what-will-it-take-to-get-to-the-next-level
[4] https://bkk19.sched.com/event/L2GG/bkk19-217-experiences-and-lessons-we-learned-using-kselftest-and-potential-improvements
[5] https://bkk19.sched.com/event/L2EQ/bkk19-212-lava-community-enabled-testing
[6] https://bkk19.sched.com/event/L2Ei/bkk19-211-harmonizing-open-source-test-definitions
[7] https://bkk19.sched.com/event/L2GV/bkk19-219-kernelci-new-generation
[8] https://bkk19.sched.com/event/L2El/bkk19-407-what-is-this-fuego-thing-and-where-is-it-going
[9] https://bkk19.sched.com/event/L2EK/bkk19-416-automating-test-results-analysis-using-neural-networks
[10] https://bkk19.sched.com/event/L2GA/bkk19-412-scheduling-in-cicd-systems
[11] https://bkk19.sched.com/event/L2FR/bkk19-409-bootloader-testing-in-lava
[12] https://bkk19.sched.com/event/Li6u/bkk19-tr08-how-to-integrate-fuego-automated-testing-tool-in-your-ci-loop

Thanks,
Dan

-- 
Linaro - Kernel Validation