[Automated-testing] Structured feeds

[Dmitry Vyukov]

On Wed, Nov 6, 2019 at 8:54 PM Han-Wen Nienhuys <hanwen at google.com> wrote:
>
> On Tue, Nov 5, 2019 at 11:02 AM Dmitry Vyukov <dvyukov at google.com> wrote:
> >
> > Eventually git-lfs (https://git-lfs.github.com) may be used to embed
> >
> > blob's right into feeds. This would allow users to fetch only the
> > blobs they are interested in. But this does not need to happen from
> > day one.
>
> I would avoid building something around git-lfs. The git upstream
> project is actively working on providing something that is less hacky
> and more reproducible.

Noted.
I mostly just captured what Konstantin pointed to. I think (1) blob
embedding is not version 1, (2) whatever we do, somebody needs to
prototype and try first.

> Also, if we're using Git to represent the feed and are thinking about
> embedding blobs,

Blobs are not about patches. Patches are small and not binary.
Blobs would be kernel binaries, test binaries, images, code dumps, etc.

> it would be much more practical to just add a copy of
> the linux kernel to the Lore repository, and introduce a commit for
> each patch. The linux kernel is about 1.5G, which is much smaller than
> the Lore archive, isn't it? You could store each patch under any of
> these branch names :
>
>   refs/patches/MESSAGE-ID
>   refs/patches/URL-ESCAPE(MESSAGE-ID)
>   refs/patches/SHA1(MESSAGE-ID)
>   refs/patches/AUTHOR/MESSAGE-ID
>
> this will lead to a large number of branches, but this is actually
> something that is being addressed in Git with reftable.

Interesting. I need to think how exactly it can be integrated as
kernel is not a single tree. Though, obviously fetching exact git tree
is very nice. But it's somewhat orthogonal to feeds and may be
provided by another specialized bot feed ("I posted your patch to git
and it's available here"), this way this will work for legacy email
patches too.

> > No work has been done on the actual form/schema of the structured
> > feeds. That's something we need to figure out working on a prototype.
> > However, good references would be git-appraise schema:
> > https://github.com/google/git-appraise/tree/master/schema
> > and gerrit schema (not sure what's a good link).
>
>
> The gerrit schema for reviews is unfortunately not documented, but it
> should be. I'll try to write down something next week, but here is the
> gist of it:
>
> Each review ("change") in Gerrit is numbered. The different revisions
> ("patchsets") of a change 12345 are stored under
>
>   refs/changes/45/12345/${PATCHSET_NUMBER}
>
> they are stored as commits to the main project, ie. if you fetch this
> ref, you can check out the proposed change.
>
> A change 12345 has its review metadata under
>
>   refs/changes/45/12345/meta
>
> The metadata is a notes branch. The commit messages on the branch hold
> global data on the change (votes, global comments). The per file
> comments are in a notemap, where the key is the SHA1 of the patchset
> the comment refers to, and the value is JSON data. The format of the
> JSON is here:
>
>  https://gerrit.googlesource.com/gerrit/+/9a6b8da5736536405da8bf5956fb3b47e322afa8/java/com/google/gerrit/server/notedb/RevisionNoteData.java#25
>
> with the meat in Comment class
>
>   https://gerrit.googlesource.com/gerrit/+/9a6b8da5736536405da8bf5956fb3b47e322afa8/java/com/google/gerrit/entities/Comment.java#33
>
> an example
>
>    {
>       "key": {
>         "uuid": "c7be1334_47885e36",
>         "filename":
> "java/com/google/gerrit/server/restapi/project/CommitsCollection.java",
>         "patchSetId": 7
>       },
>       "lineNbr": 158,
>       "author": {
>         "id": 1026112
>       },
>       "writtenOn": "2019-11-06T09:00:50Z",
>       "side": 1,
>       "message": "nit: factor this out in a variable, use
> toImmutableList as collector",
>       "range": {
>         "startLine": 156,
>         "startChar": 32,
>         "endLine": 158,
>         "endChar": 66
>       },
>       "revId": "071c601d6ee1a2a9f520415fd9efef8e00f9cf60",
>       "serverId": "173816e5-2b9a-37c3-8a2e-48639d4f1153",
>       "unresolved": true
>     },
>
> for CI type comments, we have "checks" data and robot comments (an
> extension of the previous comment), defined here:
>
> https://gerrit.googlesource.com/gerrit/+/9a6b8da5736536405da8bf5956fb3b47e322afa8/java/com/google/gerrit/entities/RobotComment.java#22
>
> here is an example of CI data that we keep:
>
>  "checks": {
>     "fmt:commitmsg-462a7efcf7234c5824393847968ddd28853aef6e": {
>       "state": "FAILED",
>       "message": "/COMMIT_MSG: subject must not end in \u0027.\u0027",
>       "started": "2019-09-13T17:12:46Z",
>       "created": "2019-09-11T17:42:40Z",
>       "updated": "2019-09-13T17:12:47Z"
>     }
>
> JSON definition:
> https://gerrit.googlesource.com/plugins/checks/+/0e609a4599d17308664e1d41c0f91447640ee9fe/java/com/google/gerrit/plugins/checks/db/NoteDbCheck.java#16

I've added a reference to this for future reference here:
https://github.com/dvyukov/kit/blob/master/doc/references.md
Thanks!