[yocto] [yocto-autobuilder] [RFC] Add support for generate bitbake error reports

[Joshua G Lock]

(Resent including the list, apologies for those receiving it again)

On Thu, 2016-06-09 at 16:23 -0500, Aníbal Limón wrote:
> Hi folks,
> 
> Currently we support to send error reports to errors.yoctoproject.org
> about failing tasks on bitbake using SendErrorReport buildstep but we
> have a lack of bitbake related errors like exceptions.
> 
> A bug exist to implement this support into Error report web [1], i'm
> working on it but for generate bitbake error reports there's a need
> of
> some process monitoring the bitbake output in this case the Yocto
> Autobuilder.
> 
> This email is for review my current implementation for generate
> bitbake
> error reports in the Autobuilder [2] next i'll try to explain how it
> works.
> 
> I aadded a BitbakeShellCommand [3] class for use in the buildsteps
> that
> executes bitbake the main objective of this class is to have common
> operations to be made in bitbake commands like create error reports
> if
> fails.
> 
> For create error reports this class add an stdio observer to look at
> bitbake output and if bitbake fails review the bitbake output for
> identify if the failure is an non-related task error [4]. If the
> observer found bitbake error creates an Error report with the
> information in the master controller.
> 
> In order to send the bitbake error to Error report web the controller
> transfer the report to the worker using a new DownloadDirectory
> implementation that i made [5], the SendErrorReport buildstep works
> on
> the worker side so it's easy to transfer the reports from master to
> worker instead of send it by master.
> 
> Finally to complete with the job of have the bitbake errors reports
> the
> Error report web changes need  (i'm working on it) to be integrated
> first in order to don't break anything.
> 
> Please review it and provide me feedback.

This would've been much easier to review as a series of patches.

After a quick read via the gitweb the series as a whole looks good.

A couple of review comments:

In a04b41d37c29191318386455d8d958ff815a3a10 "lib/buildsteps.py: Add
BitbakeLogLineObserver for BitbakeShellCommands." you have a comment

"discard line that are not errors and line that is recipe task errors"

but the lines are not actually discarded, afaict  from a cursory read
through they aren't used in the rest of the series
unless errors['bitbake'] == True? 

Could we move the error['log'].append() to after the if statement which
checks whether this is a bitbake error?

Minor nit, in caf472bc696053227825c5a102feef3e17574ba2 "lib/buildsteps:
BitbakeShellCommand add support for create error reports"
in get_error_report_bitbake_dir() you use both " and ' for strings.

Same in 40279597615c49bc4f4f067cbab937584b626671

Regards,

Joshua

> 
> Cheers,
> 	alimon
> 
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=7583
> [2]
> http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/log/?h=co
> ntrib/alimon/devel
> [3]
> http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/tree/lib/
> python2.7/site-
> packages/autobuilder/lib/buildsteps.py?h=contrib/alimon/devel#n92
> [4]
> http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/tree/lib/
> python2.7/site-
> packages/autobuilder/lib/buildsteps.py?h=contrib/alimon/devel#n53
> [5]
> http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/?h
> =contrib/alimon/devel&id=4022920bb0e56d1eef3dfe7c5e9b4699f801cdf1
>