[yocto] Building Custom Python 3 Packages

[Seilis, Aaron]

Hi,

I'm building a couple of custom Python 3 scripts+modules as part of the -native tools required to build a number of my embedded components. I'm having difficulties getting my recipe to build when I've run `devtool modify` but the build works fine in a regular build. The Python tools are stored in various Git repositories and use the typical setuptools infrastructure.

My recipe is roughly as follows:

> inherit setuptools3
>
> SUMMARY = "My tool"
> DESCRIPTION = "Description"
>
> SECTION = "devel/scripts"
> LICENSE = "CLOSED"
>
> SRC_URI = "my-URL"
> SRCREV = "git hash"
>
> S = "${WORKDIR}/git/"
> BBCLASSEXTEND = "native"

This builds fine when running Bitbake normally; the tool is built and can be used to build other components. However, when I run `devtool modify`, the ${S} variable gets set to my alternate source location but the ${B} variable remains under ${WORKDIR}. This causes the build to fail with the error:

> ERROR: mytool-native-1.3.5-r0 do_install: python3 setup.py install execution failed.
> ERROR: mytool-native-1.3.5-r0 do_install: Function failed: do_install (log file is located at /.../build/tmp/work/x86_64-linux/mytool-native/1.3.5-r0/temp/log.do_install.39172)
> ERROR: Logfile of failure stored in: /.../build/tmp/work/x86_64-linux/mytool-native/1.3.5-r0/temp/log.do_install.39172
> Log data follows:
> | DEBUG: Executing shell function do_install
> | /.../build/tmp/sysroots/x86_64-linux/usr/bin/python3-native/python3: can't open file 'setup.py': [Errno 2] No such file or directory
> | ERROR: python3 setup.py install execution failed.
> ERROR: Function failed: do_install (log file is located at /.../build/tmp/work/x86_64-linux/mytool-native/1.3.5-r0/temp/log.do_install.39172)
> ERROR: Task 0 (virtual:native:/.../sources/meta-myproject/recipes-tools/mytool/mytool_1.3.5.bb, do_install) failed with exit code '1'

Looking at the build recipes for Python, shows that the do_compile() step for Python modules (with setuptools3) is:

>distutils3_do_compile() {
>        if [ x86_64-linux != aarch64-WaveserverOS-linux ]; then
>                SYS=wcs
>        else
>                SYS=aarch64-WaveserverOS-linux
>        fi
>        STAGING_INCDIR=/localdata/projects/waveserver/build/tmp/sysroots/wcs/usr/include \
>        STAGING_LIBDIR=/localdata/projects/waveserver/build/tmp/sysroots/wcs/usr/lib \
>        BUILD_SYS=x86_64-linux HOST_SYS=${SYS} \
>        /localdata/projects/waveserver/build/tmp/sysroots/x86_64-linux/usr/bin/python3-native/python3 setup.py \
>        build  || \
>        bbfatal_log "python3 setup.py build_ext execution failed."
>}

This clearly indicates that the issue is that the build is looking for setup.py in the ${B} location, but it is only present in the ${S} location when `devtool modify` has been run. I have tried setting ${B} to ${S} explicitly in the recipe, but this doesn't result in ${B} being changed when I run `bitbake -e mytool`. I could always copy ${S} to ${B} in the recipe, but that seems a bit hack-ish.

Did I miss something or is there another way that Python builds are intended to work?

Thanks,
Aaron