[yocto] poky using native pkg_resources module
Alejandro Hernandez
alejandro.hernandez at linux.intel.com
Mon Oct 2 10:01:08 PDT 2017
Thanks Ulf, I will provide a fix soon.
Alejandro
On 09/30/2017 05:27 PM, Ulf Samuelsson wrote:
>
>
> Den 2017-09-30 kl. 21:32, skrev Alejandro Hernandez:
>> Hey Ulf,
>>
>>
>> Yes, the problem is that the bmap-tools script is trying to be
>> executed by python3 (HOST) but it should be
>>
>> executed by the native python3 we just built, so fixing the shebang
>> on the script should do it,
>>
>> please file the bug and email me the bug number so I can take care of
>> it.
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12148
>
> BR
> Ulf
>
>>
>>
>> Alejandro | aehs29
>>
>>
>> On 09/30/2017 01:31 PM, Ulf Samuelsson wrote:
>>>
>>>
>>> Den 2017-09-29 kl. 20:18, skrev Alejandro Hernandez:
>>>> Hey Ulf,
>>>>
>>>>
>>>>
>>>> On 09/29/2017 04:05 AM, Ulf Samuelsson wrote:
>>>>> It looks like the pkg_resource module is using the native
>>>>> installation
>>>>>
>>>>>
>>>>> I cloned poky early september, and could complete a build.
>>>>>
>>>>> When I clone poky today, the build fails.
>>>>>
>>>>> As part of "do_image_wic" from image_types_wic.bbclass
>>>>>
>>>>> A python program in 'bmap-tools-native' is executing
>>>>>
>>>>> "from pkg_resources import load_entry_point"
>>>>>
>>>>> grep'ing for pkg_resources in bmap-tools-native reveals:
>>>>>
>>>>> bmaptool: from pkg_resources import load_entry_point
>>>>> easy3_install: from pkg_resources import load_entry_point
>>>>> easy_install-3.5: from pkg_resources import load_entry_point
>>>>> =============================
>>>>> pkg_resources.py is not available inside poky at the moment.
>>>>>
>>>>> It is (or used to be) part of python-setuptools.
>>>>
>>>> Correct, it was and still is part of setuptools
>>>>>
>>>>> On Ubuntu, it is present in:
>>>>> "http://archive.ubuntu.com/ubuntu/pool/main/p/python-setuptools/python-setuptools_3.3.orig.tar.gz"
>>>>>
>>>>> and used in Ubuntu 14.04 and 16.04 to generate the
>>>>> "python3-pkg-resource" package.
>>>>>
>>>>> When python-setuptools is generated by Yocto, it is partly
>>>>> a class, but the python-setuptools-native recipe downloads
>>>>> "https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-26.2.7.zip"
>>>>>
>>>>
>>>> This should be setuptools-36.2.7 btw
>>>>
>>>>> which does not contain "pkg_resources.py"
>>>>
>>>> Its not partly a class, it uses a class to install the package
>>>> along with distutils.
>>>>>
>>>>> =============================
>>>>> When bmaptool imports pkg_resources, it will not find anything in the
>>>>> yocto build, but will find the native
>>>>> "/usr/lib/python3/dist-packages/pkg_resources.py" from
>>>>>
>>>>> This requires bmap-tools == 3.4, but the native bmap-tools in
>>>>> Ubuntu-14.04 is 2.5, and in Ubuntu-16.04 it is 3.2 so the build
>>>>> fails.
>>>>>
>>>>> Maybe Ubuntu-17.04 supports bmap-tools 3.4 and it will build there,
>>>>> but using the native pkg_resource is of course not a good idea?
>>>>
>>>> There is something wrong with your build although I am not sure
>>>> what it is, pkg_resources is still part of python(3)-setuptools, in
>>>> fact you can check that in several ways
>>>>
>>>> if you do a devshell on python3-setuptools-native you can see the
>>>> extracted package contains the pkg_resources folder (so it is still
>>>> part of the package)
>>> >
>>> I do: "bitbake -c devshell python3-setuptools-native"
>>> A terminal windows opens in
>>> "tmp/work/x86_64-linux/python3-setuptools-native/36.2.7-r0/setuptools-36.2.7"
>>>
>>>
>>> ls
>>> bootstrap.py easy_install.py PKG-INFO setuptools
>>> build launcher.c pkg_resources
>>> setuptools.egg-info
>>> CHANGES.rst LICENSE pytest.ini tests
>>> conftest.py MANIFEST.in README.rst tox.ini
>>> dist msvc-build-launcher.cmd setup.cfg
>>> docs pavement.py setup.py
>>>
>>> so the "pkg_resources" folder is present.
>>>
>>> cd pkg_resources ; ls -l
>>> -rw-r--r-- 1 ulf ulf 12211 Aug 2 00:40 api_tests.txt
>>> drwxrwxr-x 3 ulf ulf 4096 Sep 30 17:35 extern
>>> -rw-r--r-- 1 ulf ulf 104650 Aug 2 00:40 __init__.py
>>> -rw-r--r-- 1 ulf ulf 600 Aug 2 00:40 py31compat.py
>>> drwxrwxr-x 2 ulf ulf 4096 Sep 30 17:35 __pycache__
>>> drwxrwxr-x 2 ulf ulf 4096 Sep 30 17:35 tests
>>> drwxrwxr-x 4 ulf ulf 4096 Sep 30 17:35 _vendor
>>>
>>> so there is NO "pkg_resource.py" file inside.
>>>
>>> The statement that fails is "from pkg_resources import
>>> load_entry_point"
>>>
>>> "load_entry_point" is defined in pkg_resources/__init__.py
>>>
>>> Somewhere, something decides that
>>> "/usr/lib/python3/dist-packages/pkg_resources.py"
>>> is what we want.
>>>
>>> I have a fresh install of Ubuntu-16.04.
>>> This has received the latest package updates.
>>> I have also installed a set of packages I usually install on a machine.
>>> Not much more.
>>>
>>> environment lack anything python related.
>>>
>>> In one test, I git cloned a fresh copy of "poky".
>>> No layers added.'
>>> Insignificant changes to "local.conf", only changed some directory
>>> location
>>>
>>> Building the beaglebone, and problem occurs.
>>>
>>> If I build from poky "8b4f16a9cbbaf521461f699b7264fac2ac872581" from
>>> Mon Sep 4 11:39:24, I have no problem.
>>>
>>> Best Regards
>>> Ulf Samuelsson
>>>
>>>> then if you do a devshell after do_install, you can check the image
>>>> directory and find the setuptools.egg file, which is what
>>>> python-native uses later, if you unzip that
>>>> egg file, you can see that pkg_resources is there, so it was in
>>>> fact packaged on Yocto / by bitbake.
>>>>
>>>> Lastly, you can also do a
>>>> $ bitbake bmap-tools-native -c devshell
>>>>
>>>> you can then check that python3-native is in fact executing by doing a
>>>>
>>>> $ which python3
>>>>
>>>> that should point to python3-native (the one you just built)
>>>>
>>>> assuming you got python3-native correctly, execute python3
>>>>
>>>> once in python you can do a
>>>>
>>>>
>>>> from pkg_resources import load_entry_point # which would import the
>>>> module
>>>>
>>>> # check where the module is being loaded from
>>>> import sys
>>>> sys.modules['pkg_resources']
>>>>
>>>> That should print out the location of the imported module, and you
>>>> can see there that infact it was imported from the setuptools.egg
>>>> file you just built before, like this:
>>>>
>>>> >>> sys.modules['pkg_resources']
>>>> <module 'pkg_resources' from
>>>> '$HOME/builds/qemux86-64/tmp/work/x86_64-linux/bmap-tools-native/3.4-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/setuptools-36.2.7-py3.5.egg/pkg_resources/__init__.py'>
>>>>
>>>>
>>>>>
>>>>> Upgrading to a non-LTS Ubuntu is not something I would like to do...
>>>>> ============================================================================================
>>>>>
>>>>>
>>>>> Error report
>>>>>
>>>>> ============================================================================================
>>>>>
>>>>>
>>>>> | DEBUG: Executing shell function do_image_wic
>>>>> | wic create
>>>>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks"
>>>>> --vars
>>>>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/sysroots/beaglebone/imgdata/"
>>>>> -e "emagii-image" -o
>>>>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/"
>>>>>
>>>>> | INFO: Creating image(s)...
>>>>> |
>>>>> | INFO: The new image(s) can be found here:
>>>>> |
>>>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/emagii_beaglebone-201709290903-mmcblk.direct
>>>>>
>>>>> |
>>>>> | The following build artifacts were used to create the image(s):
>>>>> | ROOTFS_DIR:
>>>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/tmp.wic.slysr_64/rootfs_copy
>>>>>
>>>>> | BOOTIMG_DIR:
>>>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot/usr/share
>>>>>
>>>>> | KERNEL_DIR:
>>>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/deploy/images/beaglebone
>>>>>
>>>>> | NATIVE_SYSROOT:
>>>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native
>>>>>
>>>>> |
>>>>> | INFO: The image(s) were created using OE kickstart file:
>>>>> |
>>>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks
>>>>>
>>>>> | Traceback (most recent call last):
>>>>> | File
>>>>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native/usr/bin/bmaptool",
>>>>> line 6, in <module>
>>>>> | from pkg_resources import load_entry_point
>>>>> | File "/usr/lib/python3/dist-packages/pkg_resources.py", line
>>>>> 2749, in <module>
>>>>> | working_set = WorkingSet._build_master()
>>>>> | File "/usr/lib/python3/dist-packages/pkg_resources.py", line
>>>>> 444, in _build_master
>>>>> | ws.require(__requires__)
>>>>> | File "/usr/lib/python3/dist-packages/pkg_resources.py", line
>>>>> 725, in require
>>>>> | needed = self.resolve(parse_requirements(requirements))
>>>>> | File "/usr/lib/python3/dist-packages/pkg_resources.py", line
>>>>> 628, in resolve
>>>>> | raise DistributionNotFound(req)
>>>>> | pkg_resources.DistributionNotFound: bmap-tools==3.4
>>>>> | WARNING:
>>>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/run.do_image_wic.19342:1
>>>>> exit 1 from 'bmaptool create
>>>>> emagii-image-beaglebone-20170929070314.rootfs.wic -o
>>>>> emagii-image-beaglebone-20170929070314.rootfs.wic.bmap'
>>>>> | ERROR: Function failed: do_image_wic (log file is located at
>>>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/log.do_image_wic.19342)
>>>>>
>>>>> ERROR: Task
>>>>> (/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic)
>>>>> failed with exit code '1'
>>>>> NOTE: Tasks Summary: Attempted 3187 tasks of which 3186 didn't
>>>>> need to be rerun and 1 failed.
>>>>>
>>>>> Summary: 1 task failed:
>>>>>
>>>>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic
>>>>>
>>>>> Summary: There was 1 WARNING message shown.
>>>>> Summary: There was 1 ERROR message shown, returning a non-zero
>>>>> exit code.
>>>>>
>>>>>
>>>>>
>>>>> Best Regards
>>>>> Ulf Samuelsson
>>>>
>>
More information about the yocto
mailing list