[Automated-testing] test definitions shared library

[daniel.sangorrin at toshiba.co.jp]

> From: Dan Rue <dan.rue at linaro.org>
> Hi Daniel!

Hey!

> On Fri, Jul 12, 2019 at 05:05:10AM +0000, daniel.sangorrin at toshiba.co.jp
> wrote:
> > Finally, an alternative would be to use Go static binaries for the
> > parsing phase. Go would work in Fuego (on the host side) and on Linaro
> > (on the target side) unless you are using an OS or architecture not
> > supported by Go [12].
> 
> I just wanted to comment on this suggestion, because it is an experiment we
> have performed.
> 
> You may have noticed a binary named 'skipgen' checked into test-definitions.
> This was a compromise that allows us to generate skip lists based on a yaml
> definition. The 'skipgen' binaries parse yaml and based on board, branch,
> and environment, generate a flat skipfile that can be fed into ltp, kselftest,
> etc. (well actually we don't have a solution for implementing skips in the new
> 5.2+ version of kselftest yet :( )
> 
> So during a typical LTP LAVA run in LKFT, LAVA will download test-definitions
> and overlay it to the filesystem, which will make skipgen available. Then at
> runtime, skipgen will be called with for example,
> 
>     ../../bin/arm64/skipgen --board juno-r2 --branch 5.1 --environment
> production /lava-823218/0/tests/2_ltp-timers-
> tests/automated/linux/ltp/skipfile-lkft.yaml
> 
> This will generate the list of 'skips' that match those parameters.
> 
> To back up, we originally had flat skip files but we were getting killed with the
> inflexibility - keeping them in sync, copy pasting, etc, because it's a multi-
> dimensional problem.
> 
> We've since softened our reliance on skip lists, and mostly manage 'known
> issues' now instead in SQUAD (which also use yaml files to generate). At
> some point we may even be able to rip out the skipgen stuff rather than
> maintain it.
> 
> Anyway, my impressions of using Go as suggested:
> - It does solve the problem of being able to run things on target
>   without any filesystem requirements!
> - The go binaries aren't tiny, but we use upx to compress them further
>   before checking them into test-definitions, making them ~750KB each.
> - Having a separate repo and build process is kind of expensive in terms
>   of cognitive overhead. Developers have to learn Go a bit to
>   contribute. Setting up a go development environment is its own whole
>   thing.
> - We have hit bugs in go and upx. I would expect issue with more exotic
>   architectures or combinations.
> - Checking binaries into test-definitions is obviously not great,
>   especially since we need one-per-architecture. It would be better to
>   find some better way to do this in LAVA directly rather than abuse the
>   test-definitions repo.

Thanks, I guess that the main problem is then the maintenance overhead.

Thanks,
Daniel


> 
> Dan
> 
> References:
> - skipgen source: https://github.com/Linaro/skipgen
> - yaml skip file, parsed by skipgen: https://github.com/Linaro/test-
> definitions/blob/master/automated/linux/ltp/skipfile-lkft.yaml
> - skipgen binary locations (per-arch): https://github.com/Linaro/test-
> definitions/tree/master/automated/bin
> - known issues yaml files: https://github.com/Linaro/qa-reports-known-
> issues