[Automated-testing] ELC followup / Search for a testing framework for small distribution tests on device

[Westermann, Oliver]

Hey everyone,

I recently joined the ELC in Lyon and had a good chat with a few who may read this list. Only during the ELC I leaned about the automated testing summit and if I had not planned the next days already, I would probably have joined. During the interesting talks and some networking I also learned about this list, so I now have to ask a bunch of questions in the hope for answers ;-)

We (as in the company I work for) moved to linux for our embedded devices quite some time ago, but our testing hasn't. We have a test system for our full product, but due to the nature of the product ("cameras with intelligence"), there is a lot of implied requirements on the state of the device, which makes testing expensive (in terms of device- and setuptime), which results in only one, but very thorough test loop a day.

We're now trying to check our options to setup a smaller on-device test-stage before this setup. We would like to test features we would consider "distribution features": Network connectivity, IO features, updates and more. Most devs have their small scripts to do one thing or another, but we would prefer to distribute something that allows us to share all this and run it as part of the CI chain.

So I've looked into some of the pages on the Automated_Testing page of the elinux wiki and dove into a few of the frameworks, but actually I'm even more confused by LAVA, fuego and all the other tools than before. We already have a build system, we already have a CI system controlling build system. So I'm writing to you in an attempt to get a feeling for whats there and where we can contribute, and I will try to list some of our goal and ideas so maybe someone can throw me some good links (and hopefully, we can 

What we would like:
We would like to run simple tests (execute eg python & bash script) and validate the outputs
We would like to bring devices into a certain state, eg by an upgrade (we use sth with swupdate beneath)

Since I assume a lot of you have a better overview about whats out there, what sounds like our needs and where you think we could contribute, let me know :-)

Best regards and thanks to all of those who gave talks at OSS and ELC, it was a very cool (and overwhelming) time :-)

- Olli