[Automated-testing] Linaro Connect Report

Thu Apr 11 06:33:57 PDT 2019

Dan,

No worries, it probably dropped as I'm on the mailing list.
There is now a zoom meeting set up. I will write an invitation and send to
the list as soon as I've set up a wiki page with all the information.

//Maria

On Thu, 11 Apr 2019 at 14:31, Dan Rue <dan.rue at linaro.org> wrote:

> My original outgoing email had Maria cc'd but it seems that mailman
> dropped it... trying again, this time from gmail instead of mutt. Also
> adding Dave Pigott for an update on the PDU API which he agreed to
> follow-up on.
>
> On Wed, Apr 10, 2019 at 4:08 PM Dan Rue <dan.rue at linaro.org> wrote:
> >
> > Linaro Connect[0] was held last week in Bangkok. There were a lot of
> > sessions (listed below) around Linux testing, LAVA, and Fuego.
> >
> > We also held an impromptu micro-summit (minutes below), where we all
> > locked ourselves in a room together for the day and discussed LAVA,
> > Fuego, test definitions, and yes, pdudaemon.
> >
> > Finally, Maria Högberg (cc'd) has graciously arranged for a monthly
> > meeting that we can use to coordinate across projects throughout the
> > year - please send her an email if you'd like to be added. The first one
> > is scheduled for Thursday, May 9th at 13:00 UTC.
> >
> > Selected Sessions (videos should be available within a week or two at the
> > referenced URLs):
> > [1] EAS Unit Testing
> > [2] LAVA Users Forum
> > [3] KEYNOTE: Open Source QA - what will it take to get to the next level
> > [4] Experiences and lessons we learned using kselftest and potential
> improvements.
> > [5] LAVA community enabled testing
> > [6] Harmonizing open source test definitions
> > [7] KernelCI New Generation
> > [8] What is this Fuego thing and where is it going?
> > [9] Automating test results analysis using neural networks
> > [10] Scheduling in CI/CD systems
> > [11] Bootloader testing in LAVA
> > [12] How to integrate Fuego automated testing tool in your CI loop
> >
> > Micro-Summit Minutes
> >
> > Date: Wednesday, April 4, 2019
> >
> > Attendees
> > - Daniel Sangorrin (Toshiba)
> > - Tim Bird (Sony)
> > - Carlos Hernandez (TI)
> > - Kat Cosgrove (jfrog)
> > - Naresh Kamboju
> > - Antonio Tercerio
> > - Daniel Diaz
> > - Charles Oliveira
> > - Chase Qi
> > - Milosz Wasilewski
> > - Stevan Radakovic
> > - Dave Pigott
> > - Maria Högbergadding
> > - Luca Di Stefano
> > - Steve McIntyre
> > - Matt Hart
> > - Remi Duraffort
> > - Fathi Boudra
> > - Anders Roxell
> > - Dan Rue
> >
> > PDU API
> > - Wider community agreement on standardizing on pdudaemon
> > - Linaro lab doesn’t actually use pdudaemon
> > - Running pdudaemon @ Linaro
> >   - [Dave] Queueing vs running immediately
> >     - [matt] we can just add it to pdu daemon if this is a problem
> >     - [Matt] added in a PR
> > - Dave will write a proposal for power control API
> >   - There will be an abstraction layer for power control to turn a
> device on or
> >     off.
> > - [Dave] What about pressing buttons, turning on/off usb ports, etc
> >   - [Tim] let’s not deal with composition right now, and only components
> >   - [Carlos] we do need to solve the problem… once we have an interface
> for
> >     power, we need a similar interface for relays.
> >   - Combinations of things may still make it complicated. For example,
> hold a
> >     button while pressing power.
> > - Core pdu api actions: on, off, status
> > - Core relay api actions: on, off, ???
> >
> > USB control
> > - LAVA lab uses cambrionix USB hubs
> > - Also use good usb shielded cables
> > - Sony uses an open hardware board per DUT
> >
> > Fuego Architecture
> > - Tim gave a subset of his Thursday talk; see his Thursday talk for the
> full
> >   details.
> > - Jenkins based
> > - Test execution system
> >   - Steps broken into discrete phases like build, deploy, run, process,
> etc.
> >   - Board and platform management are abstracted
> > - Not serial console based; typically uses SSH on a board that’s already
> >   provisioned (imaged)
> > - Contains functional and benchmark tests
> > - Fuego is a linux distribution, distributed as a container, self
> contained
> > - Testing driven from host, using ssh connection to DUT
> > - Is container privileged?
> >   - [tim] yes
> >   - [dave] Dynamic device detection is possible without privileged mode
> > - Designed for embedded linux testing
> >   - IoT possible
> > - By default, builds tests, though LTP has a way of checking to see if
> it’s on
> >   disk and skipping the build
> >   - Can also just skip the build phase so long as the build is in the
> location
> >     that is expected on target
> > - [long discussion about skip lists, known issues, and (lack of) test
> >   documentation]
> >   - Fuego-core has skip logic inside the test folder (ltp for example)
> >   - Test-definitions has similar
> >   - Additionally, lkft uses known issues in SQUAD to annotate known
> failures
> >
> > LAVA Architecture
> > - Remi gave a brief LAVA architecture overview
> > - Components:
> >   - Lava-server
> >     - Apache2, lava-server-gunicorn, postgresql, lava-logs, lava-master
> >   - Lava-dispatcher
> >     - Lava-slave, lava-run, devices under test
> > - One lava-server, multiple dispatchers, multiple DUTs per dispatcher
> > - Lava-run is ephemeral process that communicates with DUT during a job
> run
> > - Device-type defines e.g. a raspberry pi
> > - Device defines an instance of a raspberry pi, with specifics about
> which port
> >   its plugged into, what serial port is it, etc.
> > - [daniel s] can i skip the deploy?
> >   - Yes, it’s supported
> > - [daniel s] parsing in the dispatcher, where is that done?
> >   - [steve] we recommend parsing gets done on DUT
> >   - [remi] interestedin junit, tap13 parsing in LAVA so that a parser on
> DUT
> >     isn’t necessary
> >   - [remi] goal is to not have to parse on DUT and do as much as we can
> on
> >     dispatcher
> > - Multinode jobs, used for networking, controlling lab hardware using
> lxc, etc
> > - Remi mentioned a feature to run a container as a part of a test run,
> which
> >   could help with jobs that are more complex, and eliminate a lot of
> existing
> >   multi-node jobs (making it easier)
> >
> > Running Fuego tests in Lava
> > - Fuego operates a lot like an "interactive" LAVA test
> > - Are several solutions here:  One that seems good is to use the new
> feature to
> >   run a container as part of a test.
> >
> > Running Linaro tests in Fuego
> > - Fuego already has Functional.linaro, which runs a Linaro test using
> >   testrunner.
> >
> > Test Definitions
> > Tim Bird proposing `run-test.sh` as a naming convention for the name of
> the
> > test program for each package
> > - In ptest, you can build a test package, which you can install and
> execute.
> > - More controversially, why don’t you rename all the <testname>.sh
> scripts to
> >   run-test.sh
> > Tim Bird proposing a standard variable name for the location of kernel
> config
> > path
> > - LTP implements ‘KCONFIG_PATH’
> > How do we decide such standards?
> > - Proposal @ Automated testing list
> > - Tap13 is just a website. Maybe we need something similar to publish
> these
> >   things on
> > Tim Bird on Test Names
> > - Standardizing on test names would be nice so that we can compare across
> >   projects
> > - Related to test case definition discussion
> > - A good first step is to investigate how tests are named by various
> projects,
> >   starting with LTP.
> >   - How different are test case names for LTP?
> >
> > Carlos Hernandez - Test case definition standards
> > Provide everything needed to run any test from any test framework.
> > - TGUID
> > - Test Name
> > - Description
> > - Test Execution Engine (e.g. LAVA, Fuego, VATF)
> > - Test script path
> >
> > Next Steps
> > Agreement to start a public monthly meeting. Maria will organize.
> >
> > [0] https://connect.linaro.org/
> > [1] https://bkk19.sched.com/event/L2DP/bkk19-114-eas-unit-testing
> > [2] https://bkk19.sched.com/event/L2EE/bkk19-120-lava-users-forum
> > [3]
> https://bkk19.sched.com/event/L2H8/bkk19-200k2-keynote-open-source-qa-what-will-it-take-to-get-to-the-next-level
> > [4]
> https://bkk19.sched.com/event/L2GG/bkk19-217-experiences-and-lessons-we-learned-using-kselftest-and-potential-improvements
> > [5]
> https://bkk19.sched.com/event/L2EQ/bkk19-212-lava-community-enabled-testing
> > [6]
> https://bkk19.sched.com/event/L2Ei/bkk19-211-harmonizing-open-source-test-definitions
> > [7] https://bkk19.sched.com/event/L2GV/bkk19-219-kernelci-new-generation
> > [8]
> https://bkk19.sched.com/event/L2El/bkk19-407-what-is-this-fuego-thing-and-where-is-it-going
> > [9]
> https://bkk19.sched.com/event/L2EK/bkk19-416-automating-test-results-analysis-using-neural-networks
> > [10]
> https://bkk19.sched.com/event/L2GA/bkk19-412-scheduling-in-cicd-systems
> > [11]
> https://bkk19.sched.com/event/L2FR/bkk19-409-bootloader-testing-in-lava
> > [12]
> https://bkk19.sched.com/event/Li6u/bkk19-tr08-how-to-integrate-fuego-automated-testing-tool-in-your-ci-loop
> >
> > Thanks,
> > Dan
> >
> > --
> > Linaro - Kernel Validation
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/automated-testing/attachments/20190411/a1f7e667/attachment.html>