[yocto-ab] YP Advisory Board: YP Compatible changes

[William Mills]

On 06/16/2014 12:28 PM, William Mills wrote:
>
>
> On 06/13/2014 05:45 PM, Osier-mixon, Jeffrey wrote:
>> On Fri, Jun 13, 2014 at 2:34 PM, William Mills <wmills at ti.com> wrote:
>>> Jeff,
>>>
>>> Chase has the vote from TI.
>>>
>>> However, I would still like clarity / understanding about "or remove
>>> it from
>>> the list".
>>

It has been suggested that I spell out what I would like to see.
I did this a bit in the call the other day but it is unreasonable of me 
to expect anyone to remember that.  The following is more detailed
anyway.

I would like to see something like this:

1) Yocto Project Compatibility will continue to be per Yocto Project
release, even when originations are self certifying.

2) Anyone can raise a compatibility concern.  (bugs.yoctoproject.org?)
The layer owner should be included on the problem report.  The
architecture team, lead by Richard, has sole responsibility to decide
if the issue is one that requires this process to be followed.

3) Any issue verified by the architecture team needs to be fixed before 
YP Compatible is claimed for the project release on a later YP release.

If an issue if found in MyOS 45 based on YP 1.7, then it needs to be 
fixed before MyOS based on YP 1.8 can be claimed to be YP compatible.

4) It is highly desirable (required?) to fix or document as a 
compatibility errata for the current release.

MyOS 45.1 should include a fix if it won't hurt users of MyOS.
If the fix is too disruptive it should be included as an errata notice
in someway.  Suggested workarounds should be supplied when practicle. 
Choice of resolution is left to the layer owner.

Timeframe?
    Commercially reasonable
    as soon as reasonable but not to exceed 6 months
    etc.

5) It is expected that all YP members follow these rules in good faith 
and no banding rights will ever need to be revoked once granted.  If the
need does arise, branding rights can be stripped from an organization
or project by a 2/3 majority vote of all current YP members (silver ==
1 vote, gold == 2 votes).  If branding has been revoked, the
organization or project can reapply for branding and it will be granted
provided all issues have been resolved.

********

What this does:
* Put the focus for fixes on future work
* Focus for current work is knowledge of the issue
* Ensures one picky person can not torpedo a ton of marketing work or 
invalidate physical material
* Allows an intentional bad actor to loose branding


>> If there is dispute about whether a given product or layer qualifies
>> for YP Compatible, I believe it is most fair to the rest of the
>> organizations in the project to remove the product/layer from the list
>> of YP Compatible things until the dispute is resolved. That is what we
>> would do now if someone complained; I'm just stating it as policy
>> rather than leaving it vague. Note that this has never happened, and I
>> don't expect it to happen in the future, but it makes sense to plan
>> for it as a contingency.
>>
>
> I don't think you really are understanding what I am saying or you do
> not fully understand the implications of what you are saying.
>
> If this is indeed the policy that gets implemented then you would need
> to remove poky from the list of "compatible" distros.
>
> Poky 7 / denzil / YP 1.2 had a "compatible" problem in that it mixed
> distro and bsp meta data.  This was an issue for anyone using meta-ti.
>
> Poky 8 / danny / YP 1.3 fixed this issue by seperating out the bsp meta
> data into its own layer meta-yocto-bsp.
>
> Poky 7 was never "fixed".  So your statement without my qualifications
> means that poky comes off the "compatible" list.
>
> "Fixing" Poky 7 would have been stupid.  It would have forced all user
> to adjust their layer.conf data.  This would have inconvenienced all
> existing users, those that were not using meta-ti and those that were
> using meta-ti and had already worked around the issue.  Yes TI had to
> continue to show people how to work around the issue for 12 to 18
> months until people were no longer using YP 1.2.  However it was the
> right thing to do.
>
> Taking the branding away from Poky is also stupid.  Even if that 'lack
> of branding' was just for Poky 7 / YP 1.2 it would have not been the
> right message to send.
>
> I think it is fine to demand projects fix issues going forward.  I think
> it is fine to request that they fix prior versions of the product or
> docuement the errata.
>
>>> As I said on the call, issues on prior releases may not be fixable
>>> with out
>>> effecting existing customers.  Yes I trust Richard to be reasonable
>>> but I
>>> would still like to see some general principals in writing.
>>
>> Yes, of course - the next step is to get a set of guidelines down in
>> writing so we can discuss it.
>>
>> We also do need to keep in mind that even a revocation of YP
>> Compatible status would not remove an item from the public or take it
>> out of the hands of customers. Revocation would simply mean the
>> sponsoring organization would have to remove the YP Compatible badge
>> from wherever it is being used. This is good motivation to ensure that
>> the item follows the guidelines, but the only potential penalty is in
>> marketing, not sales or engineering. It's a branding program only.
>>
>
> So if the minnowboard box had a "Yocto Project Compatible" badge and I
> found an issue in meta-minnowboard that was hard to fix w/o hurting
> your existing customers you would stop shipments on minnowboards??  Or
> print new boxes??  Or go ahead and break your existing users?
>
> If that could happen then we will all quickly learn never to use the YP
> badge on anything physical like a box etc.
>
> If organizations can't count on reasonable stability of the badge access
> to written principals then they won't use it any where that is tough to
> change and may not use it at all.  Why would TI put marketing effort and
> dollars into promoting a badge that we could loose access to because
> someone new comes into the Yocto Project.
> (We have been seeing a good bit of that BTW.)
>
> I am actually very encouraged that we are moving toward a system where
> we all get more compatible as we learn from each other best practices
> for our layers and our QA checkes.
>
> However, I am still bothered by the draconian wording on the proposals
> so far.
>
>>> Bill
>>>
>>>
>>> On 06/13/2014 02:57 PM, akuster wrote:
>>>>
>>>> Jeffery,
>>>>
>>>> Sounds good to me.
>>>>
>>>> - Armin
>>>>
>>>> -----Original Message-----
>>>> From: "Osier-mixon, Jeffrey" <jeffrey.osier-mixon at intel.com>
>>>> To: akuster at mvista <akuster at mvista.com>
>>>> Cc: yocto-ab at yoctoproject.org <yocto-ab at yoctoproject.org>
>>>> Subject: Re: [yocto-ab] YP Advisory Board: YP Compatible changes
>>>> Date: Thu, 12 Jun 2014 14:51:48 -0700
>>>>
>>>> I'm reviving this discussion, as I don't want it to stall out
>>>> completely.
>>>>
>>>> Please read this all the way through. If we can't resolve this by
>>>> email, I may call a special meeting of the Advisory Board so we can
>>>> move forward. TLDR summaries are at the top of each section.
>>>> ________________________________________________
>>>>
>>>> First: YP Compatible and Participant are branding programs.
>>>>
>>>> I want to remind everyone about the difference between a layer that is
>>>> "compatible" - i.e. one that has technical compatibility with the
>>>> project - and one that meets the criteria of our YP Compatible
>>>> program.
>>>>
>>>> YP Compatible and YP Participant are co-branding programs, not true
>>>> compliance programs. Otherwise, layers, products, or organizations
>>>> would qualify solely on meeting specific criteria, and there would be
>>>> no need for a vote.
>>>>
>>>> In addition, products can be designated YP Compatible if they are
>>>> built with YP Compatible parts, pass RP's sniff test, and are voted in
>>>> by the Advisory Board. Same goes for private layers.
>>>>
>>>> The Advisory Board can grant it to anything they want if they can
>>>> convince RP that that thing deserves it. Many items that are quite
>>>> compatible with the project do not qualify for YP Compatible status
>>>> because their sponsoring orgs are not YP members (or open source
>>>> projects).
>>>>
>>>> To reiterate - YP Compatible is about branding, period. It is not a
>>>> spec, and you can't conform to it or comply with it.
>>>>
>>>> If any of this is unclear, please bring it up for discussion now.
>>>>
>>>> I would like to immediately change the name on the website from
>>>> "Compliance Program" to "Collaborative Branding Program" in the hope
>>>> that we can partly alleviate this misconception.
>>>>
>>>> In the future, I also believe we should change the name of YP
>>>> Compatible to some other adjective, but that is for a different
>>>> discussion. Let's get through redefining the program first.
>>>>
>>>> ________________________________________________
>>>>
>>>> Second - our current system requires too much of RP's time and too
>>>> much Advisory Board time on meaningless votes.
>>>>
>>>> Right now, all YP Compatible items require new votes whenever they are
>>>> superseded, or whenever YP is released. Richard's technical
>>>> organization is required to weigh in every time, and the Advisory
>>>> Board is required to vote on every change.
>>>>
>>>> As an example, let's say YP member organization Acme Sillycon has a
>>>> BSP layer called meta-acme that they have submitted for YP Compatible.
>>>> They are a member org and RP approves of their layer, so the board
>>>> votes yes (of course) and they get YP Compatible status for YP 1.10.
>>>>
>>>> Under the current system, every time there is a meta-acme release OR a
>>>> YP release, they need to reapply and the Advisory Board needs to vote.
>>>> If the layer is released twice a year and YP is released twice a year,
>>>> that is 4 Advisory Board votes per year for this one item.
>>>>
>>>> Now imagine that Acme has a dozen layers that all need this kind of
>>>> attention. And imagine that there are 20 organizations in the project
>>>> with similar needs. The Advisory Board would do little other than cast
>>>> votes.
>>>>
>>>> These are the "rubber stamp" votes that very often come up after a
>>>> release. These votes are a pointless waste of time, especially when
>>>> anyone can complain if they see something amiss. In the program's
>>>> existence since 2012, no one has ever voted against YP Compatible
>>>> status for a layer or product that had that status in the past, and no
>>>> one has ever complained about any other member organization.
>>>>
>>>> If we were a consortium or some other organization with overflowing
>>>> funds and a thirst for bureaucracy, we could hire someone to create a
>>>> spec and then police everyone to follow that spec. We do not have
>>>> anything remotely like those kinds of resources, particularly not for
>>>> a branding program.
>>>>
>>>> We are an open source project based on mutual trust, consensus, and
>>>> shared resources that add tremendous value to every one of our
>>>> organizations, not just a few, and we use our limited funds to
>>>> everyone's benefit. Sharing the load makes us all stronger.
>>>>
>>>> ________________________________________________
>>>>
>>>> Third - here are the three goals for the proposed change, and the new
>>>> rules that comprise the change.
>>>>
>>>> I have tried to summarize the proposed new rules for YP Compatible.
>>>> There are three main goals with these changes:
>>>>
>>>> - to minimize the "rubber-stamp" Advisory Board votes required for each
>>>> release
>>>>
>>>> - to offload from the technical team the responsibility for verifying
>>>> each new release
>>>>
>>>> - to empower the sponsoring organizations with the responsibility for
>>>> maintaining their own branding while keeping overall control with RP
>>>> and the Advisory Board
>>>>
>>>> To summarize the changes:
>>>>
>>>> 0 - I will work with RP to create specific guidelines for member orgs
>>>> to follow regarding what can be considered YP Compatible and what
>>>> can't. I will also work on the existing messaging to clarify the
>>>> program better.
>>>>
>>>> 1 - Going forward, layers or products only need to be voted in once.
>>>> After that, the onus is on the sponsoring organization to make sure
>>>> the layer stays within the guidelines. YP Compatible items from
>>>> non-member orgs will keep the existing rules.
>>>>
>>>> 2 - Every time YP or the item itself is released, the sponsoring org
>>>> must fill out a form verifying that the YP Compatible item still meets
>>>> the guidelines.
>>>>
>>>> 3 - If anyone complains that a product or layer does not meet the
>>>> guidelines, RP's organization and I will investigate and recommend
>>>> changes or remove it from the list. It is incumbent on the sponsoring
>>>> organization to fix any problems that arise.
>>>>
>>>> One potential addition to these rules is to revisit all layers
>>>> periodically, e.g. annually or by some other measure. That would
>>>> negate many of the benefits that these changes are designed to effect,
>>>> though, and I do not recommend it.
>>>>
>>>> As an aside, I have added a line to each program's questionnaire about
>>>> making sure they have layers listed in the OE layer index, in
>>>> accordance with RP's wishes. I have also added instructions to the
>>>> effect that each applicant should let us know how they are using the
>>>> project and how they are visibly participating.
>>>>
>>>> ________________________________________________
>>>>
>>>> I could set up a vote to authorize these changes, but I would rather
>>>> hear from everyone before moving forward. Consensus-based
>>>> decision-making is one of the strengths of the project, so if you have
>>>> specific changes you would like to make or concerns you don't feel are
>>>> being addressed, please bring them up and follow through on the
>>>> discussion so we can get to the next step.
>>>>
>>>>
>>>> More comments inline below:
>>>>
>>>>
>>>> On Sun, Jun 1, 2014 at 8:29 PM, akuster at mvista <akuster at mvista.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> On 05/29/2014 04:58 PM, Osier-mixon, Jeffrey wrote:
>>>>>>
>>>>>>
>>>>>> This proposal is provided in preparation for discussion at Monday's
>>>>>> Advisory Board meeting.
>>>>>>
>>>>>> I have put some thinking in on YP Compatible, as it really needs
>>>>>> to be
>>>>>> streamlined and better clarified, and we need to avoid the potential
>>>>>> for many dozens of new, pointless votes. I think we can do both
>>>>>> with a
>>>>>> couple of minimal changes to the process.
>>>>>>
>>>>>> As only one example, one of our member organizations wants to create
>>>>>> many distros based on their BSP layer. These distros will come from
>>>>>> from several business units, and they want them all to have YP
>>>>>> Compatible status since they are building with components that are
>>>>>> already listed as YP Compatible.
>>>>>
>>>>>
>>>>>
>>>>> How do I know which meta-layers are compatible? The web site does not
>>>>> list
>>>>> many meta layers as being compatible like meta-selinux or
>>>>> meta-cloud-services.
>>>>
>>>>
>>>> I think this applies directly to the explanation above that YP
>>>> Compatible is a branding program, not a measure of what works with YP.
>>>> The reason meta-selinux etc. are not listed is because their
>>>> maintainers have not submitted the form to make them so, probably
>>>> because branding is not required for them.
>>>>
>>>>> It seems to me if a layer has been included in the
>>>>> Yocto repo, it implies compliance.  If that layer has been branched
>>>>> to a
>>>>> supported version, then it is compatible for that version.
>>>>
>>>>
>>>> This is a very interesting and important point, and I think it bears
>>>> repeating - the repo has nothing to do with the branding program.
>>>>
>>>> "Compliance" is a term which has no real meaning in YP. Whether a
>>>> layer is listed in the YP git repo, the layer index, or even our
>>>> downloads page has no bearing on whether the owners of that layer can
>>>> use YP branding assets. Those are completely separate issues.
>>>>
>>>> The only measure of what has YP Compatible status, and can use that
>>>> status in marketing activities, is the registrar
>>>> (https://www.yoctoproject.org/ecosystem/compliance-program-registrar).
>>>>
>>>>>> However, if we need to vote on all of
>>>>>> them plus re-vote whenever either the distro or YP bumps versions,
>>>>>> we'll have several dozen pointless votes every year to rubber-stamp
>>>>>> these things.
>>>>>>
>>>>>> There are several organizations in a similar predicament, and with
>>>>>> new
>>>>>> orgs joining YP all the time, we don't want to have to accommodate
>>>>>> dozens or hundreds of votes each year. I'm not even getting into the
>>>>>> issues in which one of the core maintainers has to examine the
>>>>>> layer/product in question and provide guidance.
>>>>>
>>>>>
>>>>> Just for clarification, are requesters supposed to include their
>>>>> layer or
>>>>> make then public?
>>>>
>>>>
>>>> Whether a layer or product is public has no bearing on YP Compatible.
>>>> RP is the final arbiter on the technical side.
>>>>
>>>> On the business side, only YP member organizations and open-source
>>>> projects (like Angstrom) qualify for this branding program currently.
>>>> We have discussed many times opening it up to non-members, but the
>>>> desire has always been expressed that this program brings value to YP
>>>> membership, so we have kept it as it is.
>>>>
>>>>>> 2 - Every time the YP version changes, or the layer or project
>>>>>> version
>>>>>> changes, the sponsoring org has to let me know (by filling out a
>>>>>> form)
>>>>>> so the website can stay current. A special form will be created for
>>>>>> this purpose. This will have to be part of the release process, and
>>>>>> for layers we can also make this an automated way to add items to the
>>>>>> Downloads page on the website.
>>>>>
>>>>>
>>>>> This seems to imply a public layer.
>>>>
>>>>
>>>> This automated system would only work for things that go through this
>>>> particular process. Private layers, products, etc. would have a
>>>> different process. The point of this process is that the sponsoring
>>>> organization needs to track whether they are YP Compatible, as the
>>>> project itself does not have the resources to do this.
>>>>
>>>>>> 3 - If anyone complains about a layer/product being out of spec, the
>>>>>> sponsoring org has to respond within a few days and either propose a
>>>>>> fix or prove that the reporter is wrong.
>>>>>> If neither of those things
>>>>>> happens, the layer/product comes off the list. It can go back on the
>>>>>> list, on probation, if the owner works it out with Richard's
>>>>>> technical
>>>>>> organization, but not before.
>>>>>>
>>>>>> This gives member orgs full responsibility for their own layers and
>>>>>> products unless someone complains, but still provides a process for
>>>>>> oversight by the YP technical team. I want to keep it as simple, and
>>>>>> as empowering, as possible. This covers all of the contingencies I
>>>>>> can
>>>>>> think of, but I'd be grateful if anyone can spot anything I left out.
>>>>>
>>>>>
>>>>> How would a user of a layer know to whom to complain to? is this
>>>>> something
>>>>> that will be required in the README?
>>>>
>>>>
>>>> RP can make that a requirement, and I think it would be a good
>>>> decision. In my opinion, every layer should have a maintainer listed
>>>> in a README or MAINTAINERS file.
>>>>
>>>>>> Once these rules are ratified, I will write up a specific document
>>>>>> for
>>>>>> all layer maintainers to use that will cover the release and YP
>>>>>> Compatible process.
>>>>>
>>>>>
>>>>>
>>>>> My 2 cents.
>>>>>
>>>>>    It seems to me there may be two paths for Yocto Compatible. If a
>>>>> layer
>>>>> resides in the Yocto repo or at some other public viewing area, it
>>>>> qualifies
>>>>> for automatic compatible renewal once approved . A private layer might
>>>>> required approval every time.
>>>>>
>>>>> MV is going through this process currently. You don't know Me, MV nor
>>>>> have
>>>>> you seen our product/layer to make a decision. Its all about trust.
>>>>> That
>>>>> seems like a harder method of approving someone.
>>>>>
>>>>> well those are my ramblings for now.
>>>>
>>>>
>>>> I don't think it makes a difference whether a layer is private or
>>>> public, but it would be interesting to discuss the difference. Keep in
>>>> mind, though, that the point of this change is to put responsibility
>>>> on the sponsoring organization for maintaining their YP Compatible
>>>> status.
>>>>
>>>> thanks all
>>>>
>>>>
>>>
>>
>>
>>