[yocto-ab] Member involvement in the core of the project

[Richard Purdie]

On Fri, 2016-09-30 at 09:31 -0400, Chris Hallinan wrote:
> My first question is: why don't we do more to enforce member
> participation?

I am leading to that question :). Part of the issue is how we'd manage
it (and who would manage it?) but things like the recipe reporting
would give some metrics at least. The who part does worry me as I don't
really need more to do so whilst I could be well qualified, I'm also
overloaded.

So how do we make that work?

>   Perhaps it should be a hard requirement of membership and badging.

It is already a membership requirement.

> Personally, since 1) I am not a developer and 2) I have many demands
> for my time, I myself am not familiar with all of the testing
> infrastructure, and I'd welcome a formal 1-hour webex-style
> presentation on the subject.

Its certainly something I think we need to raise awareness of and that
would be one way to do it. There are others at Mentor who could also
probably do with being aware of more about it (including Chris
Larson!).

> I wasn't aware of that recipe reporting system, is that basically
> just meta-oe/poky?  Very nice.

Right now its basically running for oe-core but we maintainer the
maintainers list in meta-yocto since it is something I consider to be
"Yocto Project" owned rather than OE owned.

There is nothing in the system that would stop it being used for other
layers. The UI isn't really designed to filter to a layer right now so
would probably need to be a different instance but that is the kind of
thing which can be fixed if there is demand. All the code powering it
is on git.yoctoproject.org.


> I know we at Mentor have never made use of autobuilder, etc.  I've
> had discussions internally about that, and I'm certain there is value
> there, but we have our own internal testing infrastructure which as
> you can imaging, has had quite significant investment, as I'm sure is
> true for all of the commercial software vendors and likely the semi
> manufacturers, too.  Maybe part of the discussion is how to get more
> member participation there.

I think the autobuilder gets a lot of bad press and there is a lot of
misunderstanding which leads to that. Our testing strategy has multiple
different levels:

a) recipe level ptest packages
    [equivalent to "make test" in the source]
b) sanity.bbclass global configuration testing
    [does networking work, is MACHINE set?]
c) insane.bbclass build and package level testing
    [is the binary we just cross compiled of the right arch?]
    [did configure look in the host system for something(bad)?]
d) testimage 
    [does the image we just built boot?]
    [do things work? networking came up? python works? compiler works?]
e) testsdk
    [can the sdk we just built compile things?] 
f) test esdk
    [does the extensible sdk we just built work?] 
g) oe-selftest
    [if we do X, then Y, the Z does the right things happen, tests 
     specfic configs or combinations of commands]
h) bitbake-selftest - unitests for bitbake internal APIs
    [Does the XXX fetcher within bitbake do the right things?]

All of the above works outside of the autobuilder and tests different
facets of the project and its technology. The autobuilder (buildbot)
just makes sure we build a load of different combinations of settings
and makes sure all the above forms of testing get called (except for
ptest which we still manually handle right now, much to my
frustration).

So you don't have to switch to or use autobuilder(buildbot) to make use
of other areas of testing within the project. Admittedly most of the
above is mostly run against qemu, because that is what we have
available in our public infrastructure. The QA people do run it against
real hardware too though so it can be done.

I appreciate others do have their own systems and each has value. I do
think buildbot on the core does do a phenomenal job when you look at
the size of the test matrix and the shear number of combinations which
it does on a comparatively small amount of hardware in a comparatively
reasonable time. Trying to replace it out for jenkins or move it to the
cloud or replace it with lava would likely cost us a lot of work and
not really gain us much. If we could figure out a way to share some of
the internal well abstracted pieces listed above, we'd likely gain a
lot more.

> At a minimum we should put the subject on the agenda for next month's
> face-to-face.

Agreed, I mention it now to get people thinking about it.

Cheers,

Richard