[yocto] poky using native pkg_resources module

[Alejandro Hernandez]

Hey Ulf,


Yes, the problem is that the bmap-tools script is trying to be executed 
by python3 (HOST) but it should be

executed by the native python3 we just built, so fixing the shebang on 
the script should do it,

please file the bug and email me the bug number so I can take care of it.


Alejandro | aehs29


On 09/30/2017 01:31 PM, Ulf Samuelsson wrote:
>
>
> Den 2017-09-29 kl. 20:18, skrev Alejandro Hernandez:
>> Hey Ulf,
>>
>>
>>
>> On 09/29/2017 04:05 AM, Ulf Samuelsson wrote:
>>> It looks like the pkg_resource module is using the native installation
>>>
>>>
>>> I cloned poky early september, and could complete a build.
>>>
>>> When I clone poky today, the build fails.
>>>
>>> As part of "do_image_wic" from image_types_wic.bbclass
>>>
>>> A python program in 'bmap-tools-native' is executing
>>>
>>>     "from pkg_resources import load_entry_point"
>>>
>>> grep'ing for pkg_resources in bmap-tools-native reveals:
>>>
>>> bmaptool:        from pkg_resources import load_entry_point
>>> easy3_install:        from pkg_resources import load_entry_point
>>> easy_install-3.5:    from pkg_resources import load_entry_point
>>> =============================
>>> pkg_resources.py is not available inside poky at the moment.
>>>
>>> It is (or used to be) part of python-setuptools.
>>
>> Correct, it was and still is part of setuptools
>>>
>>> On Ubuntu, it is present in: 
>>> "http://archive.ubuntu.com/ubuntu/pool/main/p/python-setuptools/python-setuptools_3.3.orig.tar.gz" 
>>>
>>> and used in Ubuntu 14.04 and 16.04 to generate the
>>>     "python3-pkg-resource" package.
>>>
>>> When python-setuptools is generated by Yocto, it is partly
>>> a class, but the python-setuptools-native recipe downloads
>>>     "https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-26.2.7.zip" 
>>>
>>
>> This should be setuptools-36.2.7 btw
>>
>>> which does not contain "pkg_resources.py"
>>
>> Its not partly a class, it uses a class to install the package along 
>> with distutils.
>>>
>>> =============================
>>> When bmaptool imports pkg_resources, it will not find anything in the
>>> yocto build, but will find the native 
>>> "/usr/lib/python3/dist-packages/pkg_resources.py" from
>>>
>>> This requires bmap-tools == 3.4, but the native bmap-tools in 
>>> Ubuntu-14.04 is 2.5, and in Ubuntu-16.04 it is 3.2 so the build fails.
>>>
>>> Maybe Ubuntu-17.04 supports bmap-tools 3.4 and it will build there,
>>> but using the native pkg_resource is of course not a good idea?
>>
>> There is something wrong with your build although I am not sure what 
>> it is, pkg_resources is still part of python(3)-setuptools, in fact 
>> you can check that in several ways
>>
>> if you do a devshell on python3-setuptools-native you can see the 
>> extracted package contains the pkg_resources folder (so it is still 
>> part of the package)
> >
> I do: "bitbake  -c devshell python3-setuptools-native"
> A terminal windows opens in 
> "tmp/work/x86_64-linux/python3-setuptools-native/36.2.7-r0/setuptools-36.2.7"
>
> ls
> bootstrap.py  easy_install.py          PKG-INFO       setuptools
> build         launcher.c               pkg_resources setuptools.egg-info
> CHANGES.rst   LICENSE                  pytest.ini     tests
> conftest.py   MANIFEST.in              README.rst     tox.ini
> dist          msvc-build-launcher.cmd  setup.cfg
> docs          pavement.py              setup.py
>
> so the "pkg_resources" folder is present.
>
> cd pkg_resources ; ls -l
> -rw-r--r-- 1 ulf ulf  12211 Aug  2 00:40 api_tests.txt
> drwxrwxr-x 3 ulf ulf   4096 Sep 30 17:35 extern
> -rw-r--r-- 1 ulf ulf 104650 Aug  2 00:40 __init__.py
> -rw-r--r-- 1 ulf ulf    600 Aug  2 00:40 py31compat.py
> drwxrwxr-x 2 ulf ulf   4096 Sep 30 17:35 __pycache__
> drwxrwxr-x 2 ulf ulf   4096 Sep 30 17:35 tests
> drwxrwxr-x 4 ulf ulf   4096 Sep 30 17:35 _vendor
>
> so there is NO "pkg_resource.py" file inside.
>
> The statement that fails is "from pkg_resources import load_entry_point"
>
> "load_entry_point" is defined in pkg_resources/__init__.py
>
> Somewhere, something decides that
> "/usr/lib/python3/dist-packages/pkg_resources.py"
> is what we want.
>
> I have a fresh install of Ubuntu-16.04.
> This has received the latest package updates.
> I have also installed a set of packages I usually install on a machine.
> Not much more.
>
> environment lack anything python related.
>
> In one test, I git cloned a fresh copy of "poky".
> No layers added.'
> Insignificant changes to "local.conf", only changed some directory 
> location
>
> Building the beaglebone, and problem occurs.
>
> If I build from poky "8b4f16a9cbbaf521461f699b7264fac2ac872581" from
> Mon Sep 4 11:39:24, I have no problem.
>
> Best Regards
> Ulf Samuelsson
>
>> then if you do a devshell after do_install, you can check the image 
>> directory and find the setuptools.egg file, which is what 
>> python-native uses later, if you unzip that
>> egg file, you can see that pkg_resources is there, so it was in fact 
>> packaged on Yocto / by bitbake.
>>
>> Lastly, you can also do a
>> $ bitbake bmap-tools-native -c devshell
>>
>> you can then check that python3-native is in fact executing by doing a
>>
>> $ which python3
>>
>> that should point to python3-native (the one you just built)
>>
>> assuming you got python3-native correctly, execute python3
>>
>> once in python you can do a
>>
>>
>> from pkg_resources import load_entry_point # which would import the 
>> module
>>
>> # check where the module is being loaded from
>> import sys
>> sys.modules['pkg_resources']
>>
>> That should print out the location of the imported module, and you 
>> can see there that infact it was imported from the setuptools.egg 
>> file you just built before, like this:
>>
>>  >>> sys.modules['pkg_resources']
>> <module 'pkg_resources' from 
>> '$HOME/builds/qemux86-64/tmp/work/x86_64-linux/bmap-tools-native/3.4-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/setuptools-36.2.7-py3.5.egg/pkg_resources/__init__.py'> 
>>
>>
>>>
>>> Upgrading to a non-LTS Ubuntu is not something I would like to do...
>>> ============================================================================================ 
>>>
>>>
>>> Error report
>>>
>>> ============================================================================================ 
>>>
>>>
>>> | DEBUG: Executing shell function do_image_wic
>>> | wic create 
>>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks" 
>>> --vars 
>>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/sysroots/beaglebone/imgdata/" 
>>> -e "emagii-image" -o 
>>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/" 
>>>
>>> | INFO: Creating image(s)...
>>> |
>>> | INFO: The new image(s) can be found here:
>>> | 
>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/emagii_beaglebone-201709290903-mmcblk.direct 
>>>
>>> |
>>> | The following build artifacts were used to create the image(s):
>>> |   ROOTFS_DIR: 
>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/tmp.wic.slysr_64/rootfs_copy 
>>>
>>> |   BOOTIMG_DIR: 
>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot/usr/share 
>>>
>>> |   KERNEL_DIR: 
>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/deploy/images/beaglebone 
>>>
>>> |   NATIVE_SYSROOT: 
>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native 
>>>
>>> |
>>> | INFO: The image(s) were created using OE kickstart file:
>>> | 
>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks 
>>>
>>> | Traceback (most recent call last):
>>> |   File 
>>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native/usr/bin/bmaptool", 
>>> line 6, in <module>
>>> |     from pkg_resources import load_entry_point
>>> |   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 
>>> 2749, in <module>
>>> |     working_set = WorkingSet._build_master()
>>> |   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 
>>> 444, in _build_master
>>> |     ws.require(__requires__)
>>> |   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 
>>> 725, in require
>>> |     needed = self.resolve(parse_requirements(requirements))
>>> |   File "/usr/lib/python3/dist-packages/pkg_resources.py", line 
>>> 628, in resolve
>>> |     raise DistributionNotFound(req)
>>> | pkg_resources.DistributionNotFound: bmap-tools==3.4
>>> | WARNING: 
>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/run.do_image_wic.19342:1 
>>> exit 1 from 'bmaptool create 
>>> emagii-image-beaglebone-20170929070314.rootfs.wic -o 
>>> emagii-image-beaglebone-20170929070314.rootfs.wic.bmap'
>>> | ERROR: Function failed: do_image_wic (log file is located at 
>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/log.do_image_wic.19342) 
>>>
>>> ERROR: Task 
>>> (/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic) 
>>> failed with exit code '1'
>>> NOTE: Tasks Summary: Attempted 3187 tasks of which 3186 didn't need 
>>> to be rerun and 1 failed.
>>>
>>> Summary: 1 task failed:
>>>
>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic 
>>>
>>> Summary: There was 1 WARNING message shown.
>>> Summary: There was 1 ERROR message shown, returning a non-zero exit 
>>> code.
>>>
>>>
>>>
>>> Best Regards
>>> Ulf Samuelsson
>>