[Automated-testing] Automated testing with LAVA

[Markus Boos]

Hi Allen


> -----Ursprüngliche Nachricht-----
> Von: Alan Bennett [mailto:alan.bennett at linaro.org]
> Gesendet: Donnerstag, 4. September 2014 19:24
> An: Boos Markus
> Cc: automated-testing at yoctoproject.org
> Betreff: Re: [Automated-testing] Automated testing with LAVA
> 
> Hi Markus;
> 
> On 4 September 2014 06:02, Markus Boos <Markus.Boos at kistler.com> wrote:
> 
> 
> 	Hi Folks
> 
> 	I'm Test Manger in a company using Yocto for several products.
> 	And we are start to setup an automated test environment for those products
> as well.
> 
> 	> -----Ursprüngliche Nachricht-----
> 	> Von: automated-testing-bounces at yoctoproject.org [mailto:automated-
> testing-
> 	> bounces at yoctoproject.org] Im Auftrag von Paul Eggleton
> 	> Gesendet: Donnerstag, 4. September 2014 13:23
> 	> An: automated-testing at yoctoproject.org
> 	> Cc: Chase Maupin; Ionut Chisanovici; Catalin Scrieciu
> 	> Betreff: [Automated-testing] Automated testing with LAVA
> 
> 	>
> 	> Hi folks,
> 	>
> 	> As some of you may know, over the last few months we've been working
> through
> 	> some exercises with LAVA attempting to see how it could be integrated
> into the Yocto
> 	> Project QA workflow. At this point I think we've reached some conclusions
> about how
> 	> it's currently working, and I'd like to schedule a meeting to discuss that
> with
> 	> interested parties some time next week (perhaps on a hangout as
> previously?)
> 	>
> 	> It also strikes me that this mailing list has gone a little quiet lately, we've
> not had a
> 	> meeting for a while and I've not really heard anything from other folks
> working on
> 	> LAVA integration - it would be good to get an update on how that is going.
> 
> 	We do not work on LAVA integration cause our test system should work with
> dll's and other nasty stuff like NI cards. So we are focusing on using robot framework
> for our purpose.
> 
> 	Furthermore we may extend the ptest feature with black/with listing
> packages and have the result available in xUnit format to merge them easily into our
> management tools.
> 	Someone interested in such extensions of the ptest feature?
> 
> 
> 
> I definitely would like to know what your plans are here.  I think the biggest value of
> any automation system is ultimately the test portfolio and I'd love to be able to run
> and trend each nightly build's ptest results.  However, based on the few ptest
> samples I looked at it's not going to be 100% straightforward.  The heavy use of
> special characters and repetition / nesting will require us to create a pretty smart
> state machine to parse through the results, and though it's not necessarily difficult, ...
> 
> 

There are several things we considered:

*Memory Footprint*
We have 256 MB flash available on our target. 
Enabling the ptest features for the yocto based packages we used increased our smallest image by more than 100% from 56MB to 115 MB.
Including the (currently not existing) ptests for our own packages would increase the size of the image further more.

Therefore we need a possibility to choose the ptests.

*Displaying Test Results*
We run our unit tests with googletest which delivers a xUnit file as a result. For those unit test we use the ptest script as a wrapper for the unit tests.
As we use Jenkins as our CI system it allows us to show the results (xUnit files) in Jenkins or put it to an ALM tool.
I know that Jenkins is not the "preferred" CI system for yocto based projects but we need to build software for windows and for microcontrollers without OS as well.

For us as a first "milestone" having the ptest-runner results in xUnit format would be enough. So I would just patch the ptest-runner script and not the ptest script from each package.

*LAVA vs. Robotframework*
As our IT only supports Windows for all employees it would become a lot of work for us to support the involved people how to setup and run an linux environment. And products based on our platform vary a lot in their in and outputs. They should be able to build up their own system tests without support from our side.

> 
> 	>
> 	> Could you let me know your availability for a meeting next week (especially
> those on
> 	> CC, but others are welcome)? I would tentatively suggest Friday UK
> evening / US
> 	> Pacific morning time - would that work?
> 	>
> 	> Thanks,
> 	> Paul
> 	>
> 	> --
> 	>
> 	> Paul Eggleton
> 	> Intel Open Source Technology Centre
> 
> 	> --
> 	> _______________________________________________
> 	> automated-testing mailing list
> 	> automated-testing at yoctoproject.org
> 	> https://lists.yoctoproject.org/listinfo/automated-testing
> 
> 	Cheers
> 	Markus
> 	--
> 	_______________________________________________
> 	automated-testing mailing list
> 	automated-testing at yoctoproject.org
> 	https://lists.yoctoproject.org/listinfo/automated-testing
> 
> 
> 
> 
> 
> --
> 
> 
> Alan Bennett, Director of Product Technology Linaro.org <http://www.linaro.org/>  │
> Open source software for ARM SoCs | Follow Linaro: Facebook
> <http://www.facebook.com/pages/Linaro>  | Twitter
> <http://twitter.com/#%21/linaroorg>  | Blog <http://www.linaro.org/linaro-blog/>
> irc: akbennett | alan.bennett at linaro.org | linaro toolchain, linaro stable kernel & linaro
> lava
> 

Best regards
Markus