[Automated-testing] Linaro Connect Report

[Dan Rue]

My original outgoing email had Maria cc'd but it seems that mailman
dropped it... trying again, this time from gmail instead of mutt. Also
adding Dave Pigott for an update on the PDU API which he agreed to
follow-up on.

On Wed, Apr 10, 2019 at 4:08 PM Dan Rue <dan.rue at linaro.org> wrote:
>
> Linaro Connect[0] was held last week in Bangkok. There were a lot of
> sessions (listed below) around Linux testing, LAVA, and Fuego.
>
> We also held an impromptu micro-summit (minutes below), where we all
> locked ourselves in a room together for the day and discussed LAVA,
> Fuego, test definitions, and yes, pdudaemon.
>
> Finally, Maria Högberg (cc'd) has graciously arranged for a monthly
> meeting that we can use to coordinate across projects throughout the
> year - please send her an email if you'd like to be added. The first one
> is scheduled for Thursday, May 9th at 13:00 UTC.
>
> Selected Sessions (videos should be available within a week or two at the
> referenced URLs):
> [1] EAS Unit Testing
> [2] LAVA Users Forum
> [3] KEYNOTE: Open Source QA - what will it take to get to the next level
> [4] Experiences and lessons we learned using kselftest and potential improvements.
> [5] LAVA community enabled testing
> [6] Harmonizing open source test definitions
> [7] KernelCI New Generation
> [8] What is this Fuego thing and where is it going?
> [9] Automating test results analysis using neural networks
> [10] Scheduling in CI/CD systems
> [11] Bootloader testing in LAVA
> [12] How to integrate Fuego automated testing tool in your CI loop
>
> Micro-Summit Minutes
>
> Date: Wednesday, April 4, 2019
>
> Attendees
> - Daniel Sangorrin (Toshiba)
> - Tim Bird (Sony)
> - Carlos Hernandez (TI)
> - Kat Cosgrove (jfrog)
> - Naresh Kamboju
> - Antonio Tercerio
> - Daniel Diaz
> - Charles Oliveira
> - Chase Qi
> - Milosz Wasilewski
> - Stevan Radakovic
> - Dave Pigott
> - Maria Högbergadding
> - Luca Di Stefano
> - Steve McIntyre
> - Matt Hart
> - Remi Duraffort
> - Fathi Boudra
> - Anders Roxell
> - Dan Rue
>
> PDU API
> - Wider community agreement on standardizing on pdudaemon
> - Linaro lab doesn’t actually use pdudaemon
> - Running pdudaemon @ Linaro
>   - [Dave] Queueing vs running immediately
>     - [matt] we can just add it to pdu daemon if this is a problem
>     - [Matt] added in a PR
> - Dave will write a proposal for power control API
>   - There will be an abstraction layer for power control to turn a device on or
>     off.
> - [Dave] What about pressing buttons, turning on/off usb ports, etc
>   - [Tim] let’s not deal with composition right now, and only components
>   - [Carlos] we do need to solve the problem… once we have an interface for
>     power, we need a similar interface for relays.
>   - Combinations of things may still make it complicated. For example, hold a
>     button while pressing power.
> - Core pdu api actions: on, off, status
> - Core relay api actions: on, off, ???
>
> USB control
> - LAVA lab uses cambrionix USB hubs
> - Also use good usb shielded cables
> - Sony uses an open hardware board per DUT
>
> Fuego Architecture
> - Tim gave a subset of his Thursday talk; see his Thursday talk for the full
>   details.
> - Jenkins based
> - Test execution system
>   - Steps broken into discrete phases like build, deploy, run, process, etc.
>   - Board and platform management are abstracted
> - Not serial console based; typically uses SSH on a board that’s already
>   provisioned (imaged)
> - Contains functional and benchmark tests
> - Fuego is a linux distribution, distributed as a container, self contained
> - Testing driven from host, using ssh connection to DUT
> - Is container privileged?
>   - [tim] yes
>   - [dave] Dynamic device detection is possible without privileged mode
> - Designed for embedded linux testing
>   - IoT possible
> - By default, builds tests, though LTP has a way of checking to see if it’s on
>   disk and skipping the build
>   - Can also just skip the build phase so long as the build is in the location
>     that is expected on target
> - [long discussion about skip lists, known issues, and (lack of) test
>   documentation]
>   - Fuego-core has skip logic inside the test folder (ltp for example)
>   - Test-definitions has similar
>   - Additionally, lkft uses known issues in SQUAD to annotate known failures
>
> LAVA Architecture
> - Remi gave a brief LAVA architecture overview
> - Components:
>   - Lava-server
>     - Apache2, lava-server-gunicorn, postgresql, lava-logs, lava-master
>   - Lava-dispatcher
>     - Lava-slave, lava-run, devices under test
> - One lava-server, multiple dispatchers, multiple DUTs per dispatcher
> - Lava-run is ephemeral process that communicates with DUT during a job run
> - Device-type defines e.g. a raspberry pi
> - Device defines an instance of a raspberry pi, with specifics about which port
>   its plugged into, what serial port is it, etc.
> - [daniel s] can i skip the deploy?
>   - Yes, it’s supported
> - [daniel s] parsing in the dispatcher, where is that done?
>   - [steve] we recommend parsing gets done on DUT
>   - [remi] interestedin junit, tap13 parsing in LAVA so that a parser on DUT
>     isn’t necessary
>   - [remi] goal is to not have to parse on DUT and do as much as we can on
>     dispatcher
> - Multinode jobs, used for networking, controlling lab hardware using lxc, etc
> - Remi mentioned a feature to run a container as a part of a test run, which
>   could help with jobs that are more complex, and eliminate a lot of existing
>   multi-node jobs (making it easier)
>
> Running Fuego tests in Lava
> - Fuego operates a lot like an "interactive" LAVA test
> - Are several solutions here:  One that seems good is to use the new feature to
>   run a container as part of a test.
>
> Running Linaro tests in Fuego
> - Fuego already has Functional.linaro, which runs a Linaro test using
>   testrunner.
>
> Test Definitions
> Tim Bird proposing `run-test.sh` as a naming convention for the name of the
> test program for each package
> - In ptest, you can build a test package, which you can install and execute.
> - More controversially, why don’t you rename all the <testname>.sh scripts to
>   run-test.sh
> Tim Bird proposing a standard variable name for the location of kernel config
> path
> - LTP implements ‘KCONFIG_PATH’
> How do we decide such standards?
> - Proposal @ Automated testing list
> - Tap13 is just a website. Maybe we need something similar to publish these
>   things on
> Tim Bird on Test Names
> - Standardizing on test names would be nice so that we can compare across
>   projects
> - Related to test case definition discussion
> - A good first step is to investigate how tests are named by various projects,
>   starting with LTP.
>   - How different are test case names for LTP?
>
> Carlos Hernandez - Test case definition standards
> Provide everything needed to run any test from any test framework.
> - TGUID
> - Test Name
> - Description
> - Test Execution Engine (e.g. LAVA, Fuego, VATF)
> - Test script path
>
> Next Steps
> Agreement to start a public monthly meeting. Maria will organize.
>
> [0] https://connect.linaro.org/
> [1] https://bkk19.sched.com/event/L2DP/bkk19-114-eas-unit-testing
> [2] https://bkk19.sched.com/event/L2EE/bkk19-120-lava-users-forum
> [3] https://bkk19.sched.com/event/L2H8/bkk19-200k2-keynote-open-source-qa-what-will-it-take-to-get-to-the-next-level
> [4] https://bkk19.sched.com/event/L2GG/bkk19-217-experiences-and-lessons-we-learned-using-kselftest-and-potential-improvements
> [5] https://bkk19.sched.com/event/L2EQ/bkk19-212-lava-community-enabled-testing
> [6] https://bkk19.sched.com/event/L2Ei/bkk19-211-harmonizing-open-source-test-definitions
> [7] https://bkk19.sched.com/event/L2GV/bkk19-219-kernelci-new-generation
> [8] https://bkk19.sched.com/event/L2El/bkk19-407-what-is-this-fuego-thing-and-where-is-it-going
> [9] https://bkk19.sched.com/event/L2EK/bkk19-416-automating-test-results-analysis-using-neural-networks
> [10] https://bkk19.sched.com/event/L2GA/bkk19-412-scheduling-in-cicd-systems
> [11] https://bkk19.sched.com/event/L2FR/bkk19-409-bootloader-testing-in-lava
> [12] https://bkk19.sched.com/event/Li6u/bkk19-tr08-how-to-integrate-fuego-automated-testing-tool-in-your-ci-loop
>
> Thanks,
> Dan
>
> --
> Linaro - Kernel Validation