[Automated-testing] Structured feeds

[Dmitry Vyukov]

On Thu, Nov 7, 2019 at 11:41 AM Daniel Axtens <dja at axtens.net> wrote:
>
> Dmitry Vyukov <dvyukov at google.com> writes:
>
> > On Wed, Nov 6, 2019 at 4:35 PM Daniel Axtens <dja at axtens.net> wrote:
> >>
> >> > As soon as we have a bridge from plain-text emails into the structured
> >> > form, we can start building everything else in the structured world.
> >> > Such bridge needs to parse new incoming emails, try to make sense out
> >> > of them (new patch, new patch version, comment, etc) and then push the
> >> > information in structured form. Then e.g. CIs can fetch info about
> >>
> >> This is an non-trivial problem, fwiw. Patchwork's email parser clocks in
> >> at almost thirteen hundred lines, and that's with the benefit of the
> >> Python standard library. It also regularly gets patched to handle
> >> changes to email systems (e.g. DMARC), changes to git (git request-pull
> >> format changed subtly in 2.14.3), the bizzare ways people send email,
> >> and so on.
> >>
> >> Patchwork does expose much of this as an API, for example for patches:
> >> https://patchwork.ozlabs.org/api/patches/?order=-id so if you want to
> >> build on that feel free. We can possibly add data to the API if that
> >> would be helpful. (Patches are always welcome too, if you don't want to
> >> wait an indeterminate amount of time.)
> >
> > Hi Daniel,
> >
> > Thanks!
> > Could you provide a link to the code?
> > Do you have a test suite for the parser (set of email samples and what
> > they should be parsed to)?
>
> Sure:
> https://github.com/getpatchwork/patchwork in particular
> https://github.com/getpatchwork/patchwork/blob/master/patchwork/parser.py and
> https://github.com/getpatchwork/patchwork/tree/master/patchwork/tests

Added here for future reference:
https://github.com/dvyukov/kit/blob/master/doc/references.md#patchwork
Thanks!