[Automated-testing] How does kernelCI use LAVA?

[Milosz Wasilewski]

On Sat, 2 Nov 2019 at 15:31, <Tim.Bird at sony.com> wrote:
>
> Milosz and/or Kevin,
>
> This is question is just to clear up my understanding of things.
>
> I came away from ATS a bit confused by some of the discussion
> after Milosz' talk about  LAVA templating.  I thought I heard Kevin
> say that this would be very handy for KernelCI, and I don't understand why.
>
> Apparently, KernelCI has to create job definition files to run boot tests,
> and this requires that the job definition file contain things like the strings
> that are passed to the bootloader?  I believe the purpose of the templating system
> is to abstract (and possibly parameterize) these things.

I think there is some slight misunderstanding in this area. Templates
don't have to contain bootloader driving commands. I won't speak for
Kevin but it should be enough to look at KernelCI current templating
system [1] and lava-test-plans to see they're pretty similar. IMHO it
makes sense to combine them and only maintain one repository. On the
other hand it might cause the tool to become very complicated and not
easy to maintain.

>
> How does KernelCI have this information?  It seems very lab and board-specific.
> Is there some API for KernelCI to retrieve parts of the board provisioning information
> from a lab, and use that to create the job definition file to submit to LAVA?
> Is there kept in the central KernelCI master, or somehow injected by the lab LAVA master?

IIUC only thing that all KernelCI labs have to have in common is board
names. LAVA currently helps with this by providing aliases. So if the
LAB has a board that KernelCI uses, but the board name is not
compatible, they can use alias. As an example, we have beaglex15
boards in LKFT lab. They're called 'x15' but KernelCI requires dtb
name which is 'am57xx-beagle-x15' [3].

>
> I'm thinking I've misunderstood something, or I'm missing a layer in LAVA
> that provides these abstractions.
>
> In order to submit LAVA jobs, wouldn't the LKFT system have the same issue?
> It has to know about board provisioning and steps in order to create a LAVA yaml
> job file for LAVA to execute.

Not really. The problem test-plans are trying to address is the
ability to run any test on any device without worrying (too much)
about your build. KernelCI uses kernel images and pre-built rootfs to
boot the board from TFT with NFS. LKFT builds full OE based images and
flashes them to eMMC (in most cases). These are different in terms of
LAVA job definition. That's what the templating helps with.

>
> Maybe this only applies to boot-time tests, because they are so closely related
> to the provisioning step?  Is this same thing required when running LTP and other
> non-boot-time tests?
>

I think the problem is related to LAVA. In the old times of LAVA v1 we
tried to hide all such details from users. But that resulted in a lot
of 'lava magic' happening behind the scenes. Since this approach
limited flexibility, in LAVA v2 the whole provisioning part is exposed
and possible to be changed in the test job. So it's a tradeoff between
flexibility and convenience. LAVA provides a pretty good set of
defaults so projects like LKFT or KernelCI don't really need to change
the deployment part at the job definition level.

[1] https://github.com/kernelci/kernelci-core/tree/master/templates
[2] https://github.com/Linaro/lava-test-plans
[3] https://lkft.validation.linaro.org/scheduler/job/897937/definition#defline41

milosz

> Thanks for any explanation or clarification you can provide.
>  -- Tim
>