[Automated-testing] Structured feeds

[Han-Wen Nienhuys]

On Tue, Nov 5, 2019 at 11:02 AM Dmitry Vyukov <dvyukov at google.com> wrote:
>
> Eventually git-lfs (https://git-lfs.github.com) may be used to embed
>
> blob's right into feeds. This would allow users to fetch only the
> blobs they are interested in. But this does not need to happen from
> day one.

I would avoid building something around git-lfs. The git upstream
project is actively working on providing something that is less hacky
and more reproducible.

Also, if we're using Git to represent the feed and are thinking about
embedding blobs, it would be much more practical to just add a copy of
the linux kernel to the Lore repository, and introduce a commit for
each patch. The linux kernel is about 1.5G, which is much smaller than
the Lore archive, isn't it? You could store each patch under any of
these branch names :

  refs/patches/MESSAGE-ID
  refs/patches/URL-ESCAPE(MESSAGE-ID)
  refs/patches/SHA1(MESSAGE-ID)
  refs/patches/AUTHOR/MESSAGE-ID

this will lead to a large number of branches, but this is actually
something that is being addressed in Git with reftable.

> No work has been done on the actual form/schema of the structured
> feeds. That's something we need to figure out working on a prototype.
> However, good references would be git-appraise schema:
> https://github.com/google/git-appraise/tree/master/schema
> and gerrit schema (not sure what's a good link).


The gerrit schema for reviews is unfortunately not documented, but it
should be. I'll try to write down something next week, but here is the
gist of it:

Each review ("change") in Gerrit is numbered. The different revisions
("patchsets") of a change 12345 are stored under

  refs/changes/45/12345/${PATCHSET_NUMBER}

they are stored as commits to the main project, ie. if you fetch this
ref, you can check out the proposed change.

A change 12345 has its review metadata under

  refs/changes/45/12345/meta

The metadata is a notes branch. The commit messages on the branch hold
global data on the change (votes, global comments). The per file
comments are in a notemap, where the key is the SHA1 of the patchset
the comment refers to, and the value is JSON data. The format of the
JSON is here:

 https://gerrit.googlesource.com/gerrit/+/9a6b8da5736536405da8bf5956fb3b47e322afa8/java/com/google/gerrit/server/notedb/RevisionNoteData.java#25

with the meat in Comment class

  https://gerrit.googlesource.com/gerrit/+/9a6b8da5736536405da8bf5956fb3b47e322afa8/java/com/google/gerrit/entities/Comment.java#33

an example

   {
      "key": {
        "uuid": "c7be1334_47885e36",
        "filename":
"java/com/google/gerrit/server/restapi/project/CommitsCollection.java",
        "patchSetId": 7
      },
      "lineNbr": 158,
      "author": {
        "id": 1026112
      },
      "writtenOn": "2019-11-06T09:00:50Z",
      "side": 1,
      "message": "nit: factor this out in a variable, use
toImmutableList as collector",
      "range": {
        "startLine": 156,
        "startChar": 32,
        "endLine": 158,
        "endChar": 66
      },
      "revId": "071c601d6ee1a2a9f520415fd9efef8e00f9cf60",
      "serverId": "173816e5-2b9a-37c3-8a2e-48639d4f1153",
      "unresolved": true
    },

for CI type comments, we have "checks" data and robot comments (an
extension of the previous comment), defined here:

https://gerrit.googlesource.com/gerrit/+/9a6b8da5736536405da8bf5956fb3b47e322afa8/java/com/google/gerrit/entities/RobotComment.java#22

here is an example of CI data that we keep:

 "checks": {
    "fmt:commitmsg-462a7efcf7234c5824393847968ddd28853aef6e": {
      "state": "FAILED",
      "message": "/COMMIT_MSG: subject must not end in \u0027.\u0027",
      "started": "2019-09-13T17:12:46Z",
      "created": "2019-09-11T17:42:40Z",
      "updated": "2019-09-13T17:12:47Z"
    }

JSON definition:
https://gerrit.googlesource.com/plugins/checks/+/0e609a4599d17308664e1d41c0f91447640ee9fe/java/com/google/gerrit/plugins/checks/db/NoteDbCheck.java#16


-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado