[yocto] [yocto-autobuilder][PATCH] bin/buildlogger: add new script to aid SWAT process
Joshua G Lock
joshua.g.lock at linux.intel.com
Mon May 9 04:52:22 PDT 2016
On Mon, 2016-05-09 at 12:42 +0100, Flanagan, Elizabeth wrote:
> A few things.
>
> On 9 May 2016 at 12:06, Joshua Lock <joshua.g.lock at intel.com> wrote:
> >
> > buildlogger will be started with the autobuilder and, when
> > correctly
> > configured, monitor the AB's JSON API for newly started builds.
> > When one is
> > detected information about the build will be posted to the wiki.
> >
> > Requires a ConfigParser (ini) style configuration file at
> > AB_BASE/etc/buildlogger.conf formatted as follows:
> Can we get a buildlogger.conf.example in AB_BASE/etc?
Sure, I'll add that in a v2.
> >
> >
> > [wikiuser]
> > username = botuser
> > password = botuserpassword
> >
> > [wiki]
> > pagetitle = BuildLog
> >
> > Signed-off-by: Joshua Lock <joshua.g.lock at intel.com>
> > ---
> > .gitignore | 2 +
> > bin/buildlogger | 273
> > ++++++++++++++++++++++++++++++++++++++++++++++++
> > yocto-start-autobuilder | 8 ++
> > yocto-stop-autobuilder | 45 ++++----
> > 4 files changed, 309 insertions(+), 19 deletions(-)
> > create mode 100755 bin/buildlogger
> >
> > diff --git a/.gitignore b/.gitignore
> > index 3f9505b..48c8a85 100644
> > --- a/.gitignore
> > +++ b/.gitignore
> > @@ -8,6 +8,7 @@
> > ###################################################
> > buildset-config
> > config/autobuilder.conf
> > +etc/buildlogger.conf
> >
> > # Everything else #
> > ###################
> > @@ -25,6 +26,7 @@ yocto-controller/controller.cfg
> > yocto-controller/state.sqlite
> > yocto-controller/twistd.log*
> > yocto-controller/buildbot.tac
> > +yocto-controller/logger.log
> > yocto-worker/build-appliance/build(newcommits)
> > yocto-worker/buildbot.tac
> > yocto-worker/janitor.log
> > diff --git a/bin/buildlogger b/bin/buildlogger
> > new file mode 100755
> > index 0000000..7b39f92
> > --- /dev/null
> > +++ b/bin/buildlogger
> > @@ -0,0 +1,273 @@
> > +#!/usr/bin/env python3
> > +'''
> > +Created on May 5, 2016
> > +
> > +__author__ = "Joshua Lock"
> > +__copyright__ = "Copyright 2016, Intel Corporation"
> > +__credits__ = ["Joshua Lock"]
> > +__license__ = "GPL"
> > +__version__ = "2.0"
> > +__maintainer__ = "Joshua Lock"
> > +__email__ = "joshua.g.lock at intel.com"
> > +'''
> > +
> > +# We'd probably benefit from using some caching, but first we'd
> > need the AB API
> > +# to include
> > +#
> > +# We can set repo url, branch & commit for a bunch of repositorys.
> > +# Do they all get built for nightly?
> > +
> > +try:
> > + import configparser
> > +except ImportError:
> > + import ConfigParser as configparser
> > +import json
> > +import os
> > +import requests
> > +import signal
> > +import sys
> > +import time
> > +
> > +abapi = "https://autobuilder.yoctoproject.org/main/json/builders/n
> > ightly/builds/_all"
> > +# Wiki editing params
> > +un = ''
> > +pw = ''
> > +wikiapi = "https://wiki.yoctoproject.org/wiki/api.php"
> > +title = ''
> > +
> > +last_logged = ''
> > +# TODO: probably shouldn't write files in the same location as the
> > script?
> > +cachefile = 'buildlogger.lastbuild'
> > +tmpfile = '/tmp/.buildlogger.pid'
> > +
> > +
> > +# Load configuration information from an ini
> > +def load_config(configfile):
> > + global un
> > + global pw
> > + global title
> > + success = False
> > +
> > + if os.path.exists(configfile):
> > + try:
> > + config = configparser.ConfigParser()
> > + config.read(configfile)
> > + un = config.get('wikiuser', 'username')
> > + pw = config.get('wikiuser', 'password')
> > + title = config.get('wiki', 'pagetitle')
> > + success = True
> > + except configparser.Error as ex:
> > + print('Failed to load buildlogger configuration with
> > error: %s' % str(ex))
> > + else:
> > + print('Config file %s does not exist, please create and
> > populate it.' % configfile)
> > +
> > + return success
> > +
> > +# we can't rely on the built in JSON parser in the requests module
> > because
> > +# the JSON we get from the wiki begins with a UTF-8 BOM which
> > chokes
> > +# json.loads().
> > +# Thus we decode the raw resonse content into a string and load
> > that into a
> > +# JSON object ourselves.
> > +#
> > +# http://en.wikipedia.org/wiki/Byte_Order_Mark
> > +# http://bugs.python.org/issue18958
> > +def parse_json(response):
> > + text = response.content.decode('utf-8-sig')
> > +
> > + return json.loads(text)
> > +
> > +
> > +# Get the current content of the BuildLog page -- to make the wiki
> > page as
> > +# useful as possible the most recent log entry should be at the
> > top, to
> > +# that end we need to edit the whole page so that we can insert
> > the new entry
> > +# after the log but before the other entries.
> > +# This method fetches the current page content, splits out the
> > blurb and
> > +# returns a pair:
> > +# 1) the blurb
> > +# 2) the current entries
> > +def wiki_get_content():
> > + params =
> > '?format=json&action=query&prop=revisions&rvprop=content&titles='
> > + req = requests.get(wikiapi+params+title)
> > + parsed = parse_json(req)
> > + pageid = sorted(parsed['query']['pages'].keys())[-1]
> > + content =
> > parsed['query']['pages'][pageid]['revisions'][0]['*']
> > + blurb, entries = content.split('==', 1)
> > + # ensure we keep only a single newline after the blurb
> > + blurb = blurb.strip() + "\n"
> > + entries = '=='+entries
> > +
> > + return blurb, entries
> > +
> > +
> > +# Login to the wiki and return cookies for the logged in session
> > +def wiki_login():
> > + payload = {
> > + 'action': 'login',
> > + 'lgname': un,
> > + 'lgpassword': pw,
> > + 'utf8': '',
> > + 'format': 'json'
> > + }
> > + req1 = requests.post(wikiapi, data=payload)
> > + parsed = parse_json(req1)
> > + login_token = parsed['login']['token']
> > +
> > + payload['lgtoken'] = login_token
> > + req2 = requests.post(wikiapi, data=payload,
> > cookies=req1.cookies)
> > +
> > + return req2.cookies.copy()
> > +
> > +
> > +# Post the new page contents *content* with a summary of the
> > action *summary*
> > +def wiki_post_page(content, summary, cookies):
> > + params =
> > '?format=json&action=query&prop=info|revisions&intoken=edit&rvprop=
> > timestamp&titles='
> > + req = requests.get(wikiapi+params+title, cookies=cookies)
> > +
> > + parsed = parse_json(req)
> > + pageid = sorted(parsed['query']['pages'].keys())[-1]
> > + edit_token = parsed['query']['pages'][pageid]['edittoken']
> > +
> > + edit_cookie = cookies.copy()
> > + edit_cookie.update(req.cookies)
> > +
> > + payload = {
> > + 'action': 'edit',
> > + 'assert': 'user',
> > + 'title': title,
> > + 'summary': summary,
> > + 'text': content,
> > + 'token': edit_token,
> > + 'utf8': '',
> > + 'format': 'json'
> > + }
> > +
> > + req = requests.post(wikiapi, data=payload,
> > cookies=edit_cookie)
> > + if not req.status_code == requests.codes.ok:
> > + print("Unexpected status code %s received when trying to
> > post entry to"
> > + "the wiki." % req.status_code)
> > + return False
> > + else:
> > + return True
> > +
> > +
> > +# Extract required info about the last build from the
> > Autobuilder's JSON API
> > +# and format it for entry into the BuildLog, along with a summary
> > of the edit
> > +def ab_last_build_to_entry(build_json, build_id):
> > + build_info = build_json[build_id]
> > + builder = build_info.get('builderName', 'Unknown builder')
> > + reason = build_info.get('reason', 'No reason given')
> > + buildid = build_info.get('number', '')
> > + buildbranch = ''
> > + chash = ''
> > + for prop in build_info.get('properties'):
> > + if prop[0] == 'branch':
> > + buildbranch = prop[1]
> > + # TODO: is it safe to assume we're building from the poky
> > repo? Or at
> > + # least only to log the poky commit hash.
> > + if prop[0] == 'commit_poky':
> > + chash = prop[1]
> > +
> > + urlfmt = 'https://autobuilder.yoctoproject.org/main/builders/%
> > s/builds/%s/'
> > + url = urlfmt % (builder, buildid)
> > + sectionfmt = '==[%s %s %s - %s %s]=='
> > + section_title = sectionfmt % (url, builder, buildid,
> > buildbranch, chash)
> > + summaryfmt = 'Adding new BuildLog entry for build %s (%s)'
> > + summary = summaryfmt % (buildid, chash)
> > + content = "* '''Build ID''' - %s\n" % chash
> > + content = content + '* ' + reason + '\n'
> > + new_entry = '%s\n%s\n' % (section_title, content)
> > +
> > + return new_entry, summary
> > +
> > +
> > +# Write the last logged build id to a file
> > +def write_last_build(buildid):
> > + with open(cachefile, 'w') as fi:
> > + fi.write(buildid)
> > +
> > +
> > +# Read last logged buildid from a file
> > +def read_last_build():
> > + last_build = ''
> > + try:
> > + with open(cachefile, 'r') as fi:
> > + last_build = fi.readline()
> > + except FileNotFoundError as ex:
> > + # A build hasn't been logged yet
> > + pass
> > + except Exception as e:
> > + print('Error reading last build %s' % str(e))
> > +
> > + return last_build
> > +
> > +
> > +def watch_for_builds(configfile):
> > + if not load_config(configfile):
> > + print('Failed to start buildlogger.')
> > + sys.exit(1)
> > + last_logged = read_last_build()
> > +
> > + while True:
> > + # wait a minute...
> > + time.sleep(60)
> > +
> > + builds = requests.get(abapi)
> > +
> > + if not builds:
> > + print("Failed to fetch Autobuilder data. Exiting.")
> > + continue
> > + try:
> > + build_json = builds.json()
> > + except Exception as e:
> > + print("Failed to decode JSON: %s" % str(e))
> > + continue
> > +
> > + last_build = sorted(build_json.keys())[-1]
> > + # If a new build is detected, post a new entry to the
> > BuildLog
> > + if last_build != last_logged:
> > + new_entry, summary =
> > ab_last_build_to_entry(build_json, last_build)
> > + blurb, entries = wiki_get_content()
> > + entries = new_entry+entries
> > + cookies = wiki_login()
> > + if wiki_post_page(blurb+entries, summary, cookies):
> > + write_last_build(last_build)
> > + last_logged = last_build
> > + print("Entry posted:\n%s\n" % new_entry)
> > + else:
> > + print("Failed to post new entry.")
> > +
> > + sys.exit(0)
> > +
> > +
> > +if __name__ == "__main__":
> > + if len(sys.argv) < 2:
> > + print('Please specify the path to the config file on the
> > command line as the first argument.')
> > + sys.exit(1)
> > +
> > + # Check to see if this is running already. If so, kill it and
> > rerun
> > + if os.path.exists(tmpfile) and os.path.isfile(tmpfile):
> > + print("A prior PID file exists. Attempting to kill.")
> > + with open(tmpfile, 'r') as f:
> > + pid=f.readline()
> > + try:
> > + os.kill(int(pid), signal.SIGKILL)
> > + # We need to sleep for a second or two just to give
> > the SIGKILL time
> > + time.sleep(2)
> > + except OSError as ex:
> > + print("""We weren't able to kill the prior
> > buildlogger. Trying again.""")
> > + pass
> > + # Check if the process that we killed is alive.
> > + try:
> > + os.kill(int(pid), 0)
> > + except OSError as ex:
> > + pass
> > + elif os.path.exists(tmpfile) and not os.path.isfile(tmpfile):
> > + raise Exception("""/tmp/.buildlogger.pid is a directory,
> > remove it to continue.""")
> > + try:
> > + os.unlink(tmpfile)
> > + except:
> > + pass
> > + with open(tmpfile, 'w') as f:
> > + f.write(str(os.getpid()))
> > +
> > + watch_for_builds(sys.argv[1])
> > diff --git a/yocto-start-autobuilder b/yocto-start-autobuilder
> > index 85b748d..f8154c1 100755
> > --- a/yocto-start-autobuilder
> > +++ b/yocto-start-autobuilder
> > @@ -72,6 +72,14 @@ if os.path.isfile(os.path.join(AB_BASE,
> > ".setupdone")):
> > os.chdir(os.path.join(AB_BASE, "yocto-controller"))
> > subprocess.call(["make", "start"])
> > os.chdir(AB_BASE)
> This should be:
>
> a. Optional and defaulting to False (something in autobuilder.conf
> like PUSH_TO_WIKI)
Sure, that makes sense.
> b. Probably only want to run this on controller/both. If you run it
> on
> workers you're going to have a lot of workers hitting the page.
You can't see it from the context, but this is 4 lines beneath an
if sys.argv[1] == "controller" or sys.argv[1] == "both":
>
> Realise, most autobuilder end users won't use this functionality, so
> yeah, let's make sure this is only run when we tell it to.
You have to configure it for it to do anything, however I agree —
there's no point in even trying to start the script unless a user has
opted in. I'll add an option to autobuilder.conf
Thanks,
Joshua
More information about the yocto
mailing list