[yocto] [Toaster] Production Instance Setup

[Rudolf J Streif]

Hi Everyone:

I am trying to set up Toaster in a production environment with Apache etc. The web server setup is ok. It is running builds where I am currently stuck.

I am using the Jethro branch from the git repo.

According to step 7 in [1] one needs to start

/var/www/toaster/poky/bitbake/lib/toaster/manage.py runbuilds

to start the runner service to dispatch builds. If there are currently no pending builds (non yet created using the web UI) then running this command immediately returns. The command gets dispatched to

poky/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py

In its command help it states:

14: help    = "Schedules and executes build requests as possible. Does not return (interrupt with Ctrl-C)"

However, that is apparently not the case because if there is not build request pending then

35:            try:
36:                # select the build environment and the request to build
37:               br = self._selectBuildRequest()
38:            except IndexError as e:
39:                logger.debug("runbuilds: No build request")
40:                return

If you use the web UI and schedule (start) a build first and then execute runbuilds, then the command eventually fails with:

2016-01-14 21:37:54,086 DEBUG lbc_shellcmmd: () git remote -v
Traceback (most recent call last):
  File "/var/www/toaster/poky/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py", line 59, in schedule
    bec.triggerBuild(br.brbitbake_set.all(), br.brlayer_set.all(), br.brvariable_set.all(), br.brtarget_set.all())
  File "/var/www/toaster/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 365, in triggerBuild
    self.setLayers(bitbake, layers, targets)
  File "/var/www/toaster/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 229, in setLayers
    for remotes in self._shellcmd("git remote -v", self.be.sourcedir).split("\n"):
  File "/var/www/toaster/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py", line 61, in _shellcmd
    p = subprocess.Popen(command, cwd = cwd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory: ''

This is because the BuildEnvironment object in the database has empty fields for Sourcedir and Builddir. One can fix that by going into the Django admin interface and enter paths into these fields of the one BuildEnvironment object that is in the database.

Once the paths are entered and the build is started again then it breaks where runbuilds apparently waits for the Bitbake server to be available:

2016-01-14 22:14:54,565 DEBUG lbc_shellcmmd: (/home/fedora/source) bash -c "source /home/fedora/source/_toaster_clones/_git___git.yoctoproject.org_poky_master/oe-init-build-env /home/fedora/build 2>&1 >toaster_server.log && bitbake --read /home/fedora/build/conf/toaster-pre.conf --postread /home/fedora/build/conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0 2>&1 >>toaster_server.log "
2016-01-14 22:14:55,540 DEBUG localhostbecontroller: shellcmd success
2016-01-14 22:14:55,540 DEBUG localhostbecontroller: Found bitbake server port 33438

2016-01-14 22:14:55,543 DEBUG localhostbecontroller: Waiting bitbake server to start

... repeated 50 times ...

2016-01-14 22:15:20,595 ERROR runbuilds: Error launching build localhostbecontroller: Bitbake server did not start in 25 seconds, aborting (Error: '

That is in the file /var/www/toaster/poky/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py at lines:

    146         while not started and retries < 50:
    147             started = _toaster_ui_started(toaster_ui_log_filepath, toaster_ui_log_filelength)
    148             import time
    149             logger.debug("localhostbecontroller: Waiting bitbake server to start")
    150             time.sleep(0.5)
    151             retries += 1

The function _toaster_ui_started() that checks if Bitbake is running is defined as:

    133         def _toaster_ui_started(filepath, filepos = 0):
    134             if not os.path.exists(filepath):
    135                 return False
    136             with open(filepath, "r") as f:
    137                 f.seek(filepos)
    138                 for line in f:
    139                     if line.startswith("NOTE: ToasterUI waiting for events"):
    140                         return True
    141             return False
 
Apparently, it is looking for a line to start with the string "NOTE: ToasterUI waiting for events" in the file toaster_ui.log which is located inside the build environment that is initialized with oe-init-build-env. I grepped the Poky code base for that string and could not find any code module that ever writes that string into any file. What the file does contain though is a line that starts with "Bitbake server started", which looks rather reasonable for the purpose of checking whether the Bitbake server has been started. I am not sure why one would need to read a file to look for a specific string to check if the Bitbake server has been started, if just before localhostbecontroller states

2016-01-14 22:14:55,540 DEBUG localhostbecontroller: Found bitbake server port 33438

from reading the file toaster_server.log. Why not connect to the port and see if the Bitbake server is running?

Just for test purposes, I changed line 139 above to:

    139                     if line.startswith("Bitbake server started"):

in which case localhostbecontroller now continues with these messages:

2016-01-14 22:21:16,008 DEBUG localhostbecontroller: Waiting bitbake server to start
2016-01-14 22:21:16,509 DEBUG localhostbecontroller: Started bitbake server
2016-01-14 22:21:16,528 DEBUG localhostbecontroller: Build launched, exiting. Follow build logs at /home/fedora/build/toaster_ui.log

where it exits. However, there is no build running. Bitbake server just sits and waits.

I wonder if anyone has succeeded in making this work and can explain to me what I am missing?

Thank you,
Rudi




[1] http://www.yoctoproject.org/docs/2.0/mega-manual/mega-manual.html#toaster-installation-steps