[yocto] poky using native pkg_resources module

[Alejandro Hernandez]

Hey Ulf,



On 09/29/2017 04:05 AM, Ulf Samuelsson wrote:
> It looks like the pkg_resource module is using the native installation
>
>
> I cloned poky early september, and could complete a build.
>
> When I clone poky today, the build fails.
>
> As part of "do_image_wic" from image_types_wic.bbclass
>
> A python program in 'bmap-tools-native' is executing
>
>     "from pkg_resources import load_entry_point"
>
> grep'ing for pkg_resources in bmap-tools-native reveals:
>
> bmaptool:        from pkg_resources import load_entry_point
> easy3_install:        from pkg_resources import load_entry_point
> easy_install-3.5:    from pkg_resources import load_entry_point
> =============================
> pkg_resources.py is not available inside poky at the moment.
>
> It is (or used to be) part of python-setuptools.

Correct, it was and still is part of setuptools
>
> On Ubuntu, it is present in: 
> "http://archive.ubuntu.com/ubuntu/pool/main/p/python-setuptools/python-setuptools_3.3.orig.tar.gz"
> and used in Ubuntu 14.04 and 16.04 to generate the
>     "python3-pkg-resource" package.
>
> When python-setuptools is generated by Yocto, it is partly
> a class, but the python-setuptools-native recipe downloads
>     "https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-26.2.7.zip" 
>

This should be setuptools-36.2.7 btw

> which does not contain "pkg_resources.py"

Its not partly a class, it uses a class to install the package along 
with distutils.
>
> =============================
> When bmaptool imports pkg_resources, it will not find anything in the
> yocto build, but will find the native 
> "/usr/lib/python3/dist-packages/pkg_resources.py" from
>
> This requires bmap-tools == 3.4, but the native bmap-tools in 
> Ubuntu-14.04 is 2.5, and in Ubuntu-16.04 it is 3.2 so the build fails.
>
> Maybe Ubuntu-17.04 supports bmap-tools 3.4 and it will build there,
> but using the native pkg_resource is of course not a good idea?

There is something wrong with your build although I am not sure what it 
is, pkg_resources is still part of python(3)-setuptools, in fact you can 
check that in several ways

if you do a devshell on python3-setuptools-native you can see the 
extracted package contains the pkg_resources folder (so it is still part 
of the package)
then if you do a devshell after do_install, you can check the image 
directory and find the setuptools.egg file, which is what python-native 
uses later, if you unzip that
egg file, you can see that pkg_resources is there, so it was in fact 
packaged on Yocto / by bitbake.

Lastly, you can also do a
$ bitbake bmap-tools-native -c devshell

you can then check that python3-native is in fact executing by doing a

$ which python3

that should point to python3-native (the one you just built)

assuming you got python3-native correctly, execute python3

once in python you can do a


from pkg_resources import load_entry_point # which would import the module

# check where the module is being loaded from
import sys
sys.modules['pkg_resources']

That should print out the location of the imported module, and you can 
see there that infact it was imported from the setuptools.egg file you 
just built before, like this:

 >>> sys.modules['pkg_resources']
<module 'pkg_resources' from 
'$HOME/builds/qemux86-64/tmp/work/x86_64-linux/bmap-tools-native/3.4-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/setuptools-36.2.7-py3.5.egg/pkg_resources/__init__.py'>

>
> Upgrading to a non-LTS Ubuntu is not something I would like to do...
> ============================================================================================ 
>
>
> Error report
>
> ============================================================================================ 
>
>
> | DEBUG: Executing shell function do_image_wic
> | wic create 
> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks" 
> --vars 
> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/sysroots/beaglebone/imgdata/" 
> -e "emagii-image" -o 
> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/"
> | INFO: Creating image(s)...
> |
> | INFO: The new image(s) can be found here:
> | 
> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/emagii_beaglebone-201709290903-mmcblk.direct
> |
> | The following build artifacts were used to create the image(s):
> |   ROOTFS_DIR: 
> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/tmp.wic.slysr_64/rootfs_copy
> |   BOOTIMG_DIR: 
> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot/usr/share
> |   KERNEL_DIR: 
> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/deploy/images/beaglebone
> |   NATIVE_SYSROOT: 
> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native
> |
> | INFO: The image(s) were created using OE kickstart file:
> | 
> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks
> | Traceback (most recent call last):
> |   File 
> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native/usr/bin/bmaptool", 
> line 6, in <module>
> |     from pkg_resources import load_entry_point
> |   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2749, 
> in <module>
> |     working_set = WorkingSet._build_master()
> |   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 444, 
> in _build_master
> |     ws.require(__requires__)
> |   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 725, 
> in require
> |     needed = self.resolve(parse_requirements(requirements))
> |   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 628, 
> in resolve
> |     raise DistributionNotFound(req)
> | pkg_resources.DistributionNotFound: bmap-tools==3.4
> | WARNING: 
> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/run.do_image_wic.19342:1 
> exit 1 from 'bmaptool create 
> emagii-image-beaglebone-20170929070314.rootfs.wic -o 
> emagii-image-beaglebone-20170929070314.rootfs.wic.bmap'
> | ERROR: Function failed: do_image_wic (log file is located at 
> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/log.do_image_wic.19342)
> ERROR: Task 
> (/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic) 
> failed with exit code '1'
> NOTE: Tasks Summary: Attempted 3187 tasks of which 3186 didn't need to 
> be rerun and 1 failed.
>
> Summary: 1 task failed:
>
> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic 
>
> Summary: There was 1 WARNING message shown.
> Summary: There was 1 ERROR message shown, returning a non-zero exit code.
>
>
>
> Best Regards
> Ulf Samuelsson