[yocto] poky using native pkg_resources module
Ulf Samuelsson
yocto at emagii.com
Sat Sep 30 11:31:19 PDT 2017
Den 2017-09-29 kl. 20:18, skrev Alejandro Hernandez:
> Hey Ulf,
>
>
>
> On 09/29/2017 04:05 AM, Ulf Samuelsson wrote:
>> It looks like the pkg_resource module is using the native installation
>>
>>
>> I cloned poky early september, and could complete a build.
>>
>> When I clone poky today, the build fails.
>>
>> As part of "do_image_wic" from image_types_wic.bbclass
>>
>> A python program in 'bmap-tools-native' is executing
>>
>> "from pkg_resources import load_entry_point"
>>
>> grep'ing for pkg_resources in bmap-tools-native reveals:
>>
>> bmaptool: from pkg_resources import load_entry_point
>> easy3_install: from pkg_resources import load_entry_point
>> easy_install-3.5: from pkg_resources import load_entry_point
>> =============================
>> pkg_resources.py is not available inside poky at the moment.
>>
>> It is (or used to be) part of python-setuptools.
>
> Correct, it was and still is part of setuptools
>>
>> On Ubuntu, it is present in:
>> "http://archive.ubuntu.com/ubuntu/pool/main/p/python-setuptools/python-setuptools_3.3.orig.tar.gz"
>>
>> and used in Ubuntu 14.04 and 16.04 to generate the
>> "python3-pkg-resource" package.
>>
>> When python-setuptools is generated by Yocto, it is partly
>> a class, but the python-setuptools-native recipe downloads
>> "https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-26.2.7.zip"
>>
>
> This should be setuptools-36.2.7 btw
>
>> which does not contain "pkg_resources.py"
>
> Its not partly a class, it uses a class to install the package along
> with distutils.
>>
>> =============================
>> When bmaptool imports pkg_resources, it will not find anything in the
>> yocto build, but will find the native
>> "/usr/lib/python3/dist-packages/pkg_resources.py" from
>>
>> This requires bmap-tools == 3.4, but the native bmap-tools in
>> Ubuntu-14.04 is 2.5, and in Ubuntu-16.04 it is 3.2 so the build fails.
>>
>> Maybe Ubuntu-17.04 supports bmap-tools 3.4 and it will build there,
>> but using the native pkg_resource is of course not a good idea?
>
> There is something wrong with your build although I am not sure what it
> is, pkg_resources is still part of python(3)-setuptools, in fact you can
> check that in several ways
>
> if you do a devshell on python3-setuptools-native you can see the
> extracted package contains the pkg_resources folder (so it is still part
> of the package)
>
I do: "bitbake -c devshell python3-setuptools-native"
A terminal windows opens in
"tmp/work/x86_64-linux/python3-setuptools-native/36.2.7-r0/setuptools-36.2.7"
ls
bootstrap.py easy_install.py PKG-INFO setuptools
build launcher.c pkg_resources setuptools.egg-info
CHANGES.rst LICENSE pytest.ini tests
conftest.py MANIFEST.in README.rst tox.ini
dist msvc-build-launcher.cmd setup.cfg
docs pavement.py setup.py
so the "pkg_resources" folder is present.
cd pkg_resources ; ls -l
-rw-r--r-- 1 ulf ulf 12211 Aug 2 00:40 api_tests.txt
drwxrwxr-x 3 ulf ulf 4096 Sep 30 17:35 extern
-rw-r--r-- 1 ulf ulf 104650 Aug 2 00:40 __init__.py
-rw-r--r-- 1 ulf ulf 600 Aug 2 00:40 py31compat.py
drwxrwxr-x 2 ulf ulf 4096 Sep 30 17:35 __pycache__
drwxrwxr-x 2 ulf ulf 4096 Sep 30 17:35 tests
drwxrwxr-x 4 ulf ulf 4096 Sep 30 17:35 _vendor
so there is NO "pkg_resource.py" file inside.
The statement that fails is "from pkg_resources import load_entry_point"
"load_entry_point" is defined in pkg_resources/__init__.py
Somewhere, something decides that
"/usr/lib/python3/dist-packages/pkg_resources.py"
is what we want.
I have a fresh install of Ubuntu-16.04.
This has received the latest package updates.
I have also installed a set of packages I usually install on a machine.
Not much more.
environment lack anything python related.
In one test, I git cloned a fresh copy of "poky".
No layers added.'
Insignificant changes to "local.conf", only changed some directory location
Building the beaglebone, and problem occurs.
If I build from poky "8b4f16a9cbbaf521461f699b7264fac2ac872581" from
Mon Sep 4 11:39:24, I have no problem.
Best Regards
Ulf Samuelsson
> then if you do a devshell after do_install, you can check the image
> directory and find the setuptools.egg file, which is what python-native
> uses later, if you unzip that
> egg file, you can see that pkg_resources is there, so it was in fact
> packaged on Yocto / by bitbake.
>
> Lastly, you can also do a
> $ bitbake bmap-tools-native -c devshell
>
> you can then check that python3-native is in fact executing by doing a
>
> $ which python3
>
> that should point to python3-native (the one you just built)
>
> assuming you got python3-native correctly, execute python3
>
> once in python you can do a
>
>
> from pkg_resources import load_entry_point # which would import the module
>
> # check where the module is being loaded from
> import sys
> sys.modules['pkg_resources']
>
> That should print out the location of the imported module, and you can
> see there that infact it was imported from the setuptools.egg file you
> just built before, like this:
>
> >>> sys.modules['pkg_resources']
> <module 'pkg_resources' from
> '$HOME/builds/qemux86-64/tmp/work/x86_64-linux/bmap-tools-native/3.4-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/setuptools-36.2.7-py3.5.egg/pkg_resources/__init__.py'>
>
>
>>
>> Upgrading to a non-LTS Ubuntu is not something I would like to do...
>> ============================================================================================
>>
>>
>> Error report
>>
>> ============================================================================================
>>
>>
>> | DEBUG: Executing shell function do_image_wic
>> | wic create
>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks"
>> --vars
>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/sysroots/beaglebone/imgdata/"
>> -e "emagii-image" -o
>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/"
>>
>> | INFO: Creating image(s)...
>> |
>> | INFO: The new image(s) can be found here:
>> |
>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/emagii_beaglebone-201709290903-mmcblk.direct
>>
>> |
>> | The following build artifacts were used to create the image(s):
>> | ROOTFS_DIR:
>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/deploy-emagii-image-image-complete/emagii-image-beaglebone-20170929070314/tmp.wic.slysr_64/rootfs_copy
>>
>> | BOOTIMG_DIR:
>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot/usr/share
>>
>> | KERNEL_DIR:
>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/deploy/images/beaglebone
>>
>> | NATIVE_SYSROOT:
>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native
>>
>> |
>> | INFO: The image(s) were created using OE kickstart file:
>> |
>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/wic/emagii_beaglebone.wks
>>
>> | Traceback (most recent call last):
>> | File
>> "/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/recipe-sysroot-native/usr/bin/bmaptool",
>> line 6, in <module>
>> | from pkg_resources import load_entry_point
>> | File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2749,
>> in <module>
>> | working_set = WorkingSet._build_master()
>> | File "/usr/lib/python3/dist-packages/pkg_resources.py", line 444,
>> in _build_master
>> | ws.require(__requires__)
>> | File "/usr/lib/python3/dist-packages/pkg_resources.py", line 725,
>> in require
>> | needed = self.resolve(parse_requirements(requirements))
>> | File "/usr/lib/python3/dist-packages/pkg_resources.py", line 628,
>> in resolve
>> | raise DistributionNotFound(req)
>> | pkg_resources.DistributionNotFound: bmap-tools==3.4
>> | WARNING:
>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/run.do_image_wic.19342:1
>> exit 1 from 'bmaptool create
>> emagii-image-beaglebone-20170929070314.rootfs.wic -o
>> emagii-image-beaglebone-20170929070314.rootfs.wic.bmap'
>> | ERROR: Function failed: do_image_wic (log file is located at
>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/tmp/work/beaglebone-poky-linux-gnueabi/emagii-image/1.0-r0/temp/log.do_image_wic.19342)
>>
>> ERROR: Task
>> (/home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic)
>> failed with exit code '1'
>> NOTE: Tasks Summary: Attempted 3187 tasks of which 3186 didn't need to
>> be rerun and 1 failed.
>>
>> Summary: 1 task failed:
>>
>> /home/ulf/Kunder/eMagii/Disk/yocto-arm/poky-arm/build/../meta-emagii/recipes-images/images/emagii-image.bb:do_image_wic
>>
>> Summary: There was 1 WARNING message shown.
>> Summary: There was 1 ERROR message shown, returning a non-zero exit code.
>>
>>
>>
>> Best Regards
>> Ulf Samuelsson
>
More information about the yocto
mailing list