[yocto] Cross-compiling python fails with new custom tuning
Michael Habibi
mikehabibi at gmail.com
Mon Nov 30 15:12:49 PST 2015
OK I've dug into this further today, and here are some findings:
With my patch (which is similar to yours), it fails do_compile/package_qa
because of poisoned system directories during the compilation steps. This
approach seems to be a bust. Originally when I tested it, I hacked up the
Makefile and only re-ran do_install onward, and didn't re-run do_compile
(because I had to create the patch). Once I created the patch and re-ran
from clean, I got some errors.
It appears the only reason it is using that directory in PYTHONPATH at all
(the one generated from the shell command and pybuilddir.txt) is for a
potential sysconfig file in there. I tracked it down to a comment in a bug
on Python's bug tracker:
http://bugs.python.org/issue15484#msg180577
If we change that directory, it will pull in the sysconfig settings from
the native Python configure (probably defaults to native sysconfig settings
because the PATH doesn't include the overrides), and then do some really
bad things. The question then became: in the unmodified build, why does
do_install fail but do_compile doesn't? They both seem to be using
sharedmods recipe which uses $PYTHONPATH. The answer is that when we run
do_compile the first time around, the offending *.so's are not generated
yet, and thus are not included by python-native. The second time around (in
do_install), the *.so's are in the path and so python-native will use them
and override its own.
My latest experiment is to remove that location altogether from PYTHONPATH
and see if it works (neither the shell-generated directory in python-cross,
nor the lib-dynload directory from python-native). The compile works the
first time through despite there being no sysconfig data, so I'm wondering
if the install step will also work without finding any data. python-native
should continue to work with its internal settings to find modules and
libraries.
To be continued tomorrow, I'm sure...
On Mon, Nov 30, 2015 at 11:37 AM, Michael Habibi <mikehabibi at gmail.com>
wrote:
> Mark, I am continuing to look at this. I hope you don't mind if I keep
> updating this thread with my investigation.
>
> First, I figured out my confusion. I was associating the wrong Makefile
> command with the illegal instruction. It was actually the command under the
> recipe for 'sharedmods', but it was silenced in the Makefile so I assumed
> it was another line. You're right that the path with the *.so's is the
> offending path. I've scoured online patches and can't understand how this
> mechanism works outside of Yocto. I figure people trying to compile for ARM
> would have similar issues. There seems to be a fundamental problem with how
> Python sets up PYTHONPATH using the pybuilddir file and $abs_builddir.
> However, other people seem to have successfully built python for other
> architectures, so I'm still confused.
>
> I hardcoded a fix for PYTHONPATH to use the host lib-dynload directory,
> and the python recipe successfully passed. Can you elaborate on the issues
> you were having with your patch? The only difference is that I maintained
> the other two directories in $PYTHONPATH,
> e.g. $(srcdir)/Lib:$(srcdir)/Lib/plat-$(MACHDEP), and only replaced the
> offending one.
>
> On Wed, Nov 25, 2015 at 4:58 PM, Michael Habibi <mikehabibi at gmail.com>
> wrote:
>
>> Mark,
>>
>> I ran the same command, including the offending PYTHONPATH, from the
>> shell without error. Any reason why that would work, despite it not working
>> from within the Makefile? I was thinking it was another environment
>> variable that was causing the issue (one that's not set in my bash
>> environment by default). My $PATH didn't have python-native so I called it
>> using an absolute path, but otherwise it's the same command run from the
>> Makefile (parsed, of course):
>>
>> bash$
>> _PYTHON_PROJECT_BASE=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build
>> _PYTHON_HOST_PLATFORM=linux2-x86_64
>> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib/plat-linux2
>> /projects/yocto-git/build/tmp/sysroots/x86_64-linux/usr/bin/python-native/python2.7
>> -S -m sysconfig --generate-posix-vars
>> bash$ ls pybuilddir.txt
>> pybuilddir.txt
>>
>> Maybe I'm missing something.
>>
>>
>> On Wed, Nov 25, 2015 at 4:38 PM, Mark Hatle <mark.hatle at windriver.com>
>> wrote:
>>
>>> On 11/25/15 10:11 AM, Michael Habibi wrote:
>>> > Well, I'm wrong yet again. A 'which python2.7' shows that it's
>>> pointing to a
>>> > native version of Python:
>>> >
>>> > *| which python2.7*
>>> > *|
>>> >
>>> /projects/yocto-git/build/tmp/sysroots/x86_64-linux/usr/bin/python-native/python2.7*
>>> > |
>>> >
>>> _PYTHON_PROJECT_BASE=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build
>>> > _PYTHON_HOST_PLATFORM=linux2-x86_64
>>> >
>>> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib/plat-linux2
>>> > python2.7 -S -m sysconfig --generate-posix-vars ;\
>>> > | if test $? -ne 0 ; then \
>>> > | echo "generate-posix-vars failed" ; \
>>> > | rm -f ./pybuilddir.txt ; \
>>> > | exit 1 ; \
>>> > | fi
>>> > | Illegal instruction (core dumped)
>>> > | make: *** [sharedmods] Error 132
>>> >
>>> > I guess one of the environmental variables being passed in is screwing
>>> up what
>>> > it's doing. Unfortunately I don't know enough about the inner workings
>>> of Python
>>> > to be of much help moving forward. I gave it my best shot!
>>>
>>> The problem I tracked down was:
>>>
>>>
>>> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7
>>>
>>> This is full of .so objects when loaded cause the illegal instruction.
>>>
>>> I came up with a patch that I thought was going to fix it, but it's
>>> triggered
>>> other failures.
>>>
>>>
>>> http://lists.openembedded.org/pipermail/openembedded-core/2015-November/113341.html
>>>
>>> Just to be clear.. this does NOT work properly in many cases. But might
>>> give
>>> you or someone else a clue as to how to possibly fix it.
>>>
>>> --Mark
>>>
>>> >
>>> > On Wed, Nov 25, 2015 at 10:01 AM, Michael Habibi <mikehabibi at gmail.com
>>> > <mailto:mikehabibi at gmail.com>> wrote:
>>> >
>>> > Ah sorry, I misspoke. I walked through the Makefile and configure
>>> scripts a
>>> > bit more, and $(PYTHON_FOR_BUILD) should in fact be a host version
>>> of
>>> > Python. It seems it's not being configured correctly. Either the
>>> path is
>>> > setup wrong and python2.7 is not pointing to the right version, or
>>> python2.7
>>> > needs to be pointing to the absolute path instead of relying on
>>> $PATH.
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Nov 25, 2015 at 9:45 AM, Michael Habibi <
>>> mikehabibi at gmail.com
>>> > <mailto:mikehabibi at gmail.com>> wrote:
>>> >
>>> > For what it's worth, this is the offending portion of
>>> Makefile.pre:452:
>>> >
>>> > # Create build directory and generate the sysconfig build-time
>>> data there.
>>> > # pybuilddir.txt contains the name of the build dir and is
>>> used for
>>> > # sys.path fixup -- see Modules/getpath.c.
>>> > # Since this step runs before shared modules are built, try to
>>> avoid
>>> > bootstrap
>>> > # problems by creating a dummy pybuilddir.txt just to allow
>>> interpreter
>>> > # initialization to succeed. It will be overwritten by
>>> generate-posix-vars
>>> > # or removed in case of failure.
>>> > pybuilddir.txt: $(BUILDPYTHON)
>>> > → @echo "none" > ./pybuilddir.txt
>>> > → $(RUNSHARED) $(PYTHON_FOR_BUILD) -S -m sysconfig
>>> --generate-posix-vars ;\
>>> > → if test $$? -ne 0 ; then \
>>> > → → echo "generate-posix-vars failed" ; \
>>> > → → rm -f ./pybuilddir.txt ; \
>>> > → → exit 1 ; \
>>> > → fi
>>> >
>>> > I don't know enough about the Python build to understand what
>>> it's
>>> > trying to do, but perhaps replacing PYTHON_FOR_BUILD with
>>> HOSTPYTHON may
>>> > be acceptable?
>>> >
>>> > I'm surprised this seems to work for other people, since this
>>> should be
>>> > failing for anyone using Python on a target platform different
>>> from
>>> > their host.
>>> >
>>> > On Wed, Nov 25, 2015 at 9:02 AM, Mark Hatle <
>>> mark.hatle at windriver.com
>>> > <mailto:mark.hatle at windriver.com>> wrote:
>>> >
>>> > On 11/24/15 3:23 PM, Mark Hatle wrote:
>>> > > My guess is that there is a bug in the python
>>> integration where
>>> > it's not
>>> > > realizing the host and target are different systems, so
>>> it's
>>> > trying to run a
>>> > > target program on your host. Your host isn't haswell,
>>> so...
>>> > Illegal Instruction
>>> > > -- and the system stops.
>>> >
>>> > Just an FYI, I hit the same problem today. I suspect it
>>> is the host
>>> > trying to
>>> > run target software and I'm looking into it.
>>> >
>>> > --Mark
>>> >
>>> > > The alternative would be something is running QEMU to
>>> execute a
>>> > target binary
>>> > > and QEMU doesn't have instruction support -- but that
>>> doesn't look
>>> > like the case
>>> > > here.
>>> > >
>>> > > --Mark
>>> > >
>>> > > On 11/24/15 3:06 PM, Michael Habibi wrote:
>>> > >> All,
>>> > >>
>>> > >> I added a new machine definition with tuning parameters
>>> for haswell
>>> > >> microarchitectures (basically just duplicated corei7
>>> but tuned it
>>> > for haswell).
>>> > >> This seems to work correctly, but failed when running
>>> do_install
>>> > for python
>>> > >> toward the end of the build process. I am running with
>>> the Yocto
>>> > 2.0 framework,
>>> > >> with very minimal changes to create a new distribution
>>> and
>>> > machine for our
>>> > >> custom embedded device. Note I had this distro
>>> configuration
>>> > working before, and
>>> > >> the only difference is I added a new machine with this
>>> tuning.
>>> > >>
>>> > >> I believe the issue is because, as part of the
>>> do_install step
>>> > for Python, it
>>> > >> attempts to run python on the local host, despite being
>>> > cross-compiled. However,
>>> > >> it is tuned for a processor that my host machine
>>> doesn't run, so
>>> > I get an
>>> > >> instruction exception.
>>> > >>
>>> > >> Did I do something weird? I figure python would be
>>> configured for
>>> > >> cross-compiling and therefore wouldn't try to run the
>>> target
>>> > version on the
>>> > >> host, even for sanity tests. Here is the output of the
>>> failure:
>>> > >>
>>> > >> $ tail -20
>>> > >>
>>> >
>>> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/temp/log.do_install.17258
>>> > >>
>>> > >> x86_64-diags-linux-ar rc libpython2.7.a
>>> Modules/threadmodule.o
>>> > >> Modules/signalmodule.o Modules/posixmodule.o
>>> Modules/errnomodule.o
>>> > >> Modules/pwdmodule.o Modules/_sre.o
>>> Modules/_codecsmodule.o
>>> > >> Modules/_weakref.o Modules/zipimport.o
>>> Modules/symtablemodule.o
>>> > >> Modules/md5module.o Modules/md5.o Modules/xxsubtype.o
>>> > >> x86_64-diags-linux-ranlib libpython2.7.a
>>> > >> Modules/posixmodule.o: In function `posix_tmpnam':
>>> > >>
>>> >
>>> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Modules/posixmodule.c:7575:
>>> > >> warning: the use of `tmpnam_r' is dangerous, better use
>>> `mkstemp'
>>> > >> Modules/posixmodule.o: In function `posix_tempnam':
>>> > >>
>>> >
>>> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Modules/posixmodule.c:7522:
>>> > >> warning: the use of `tempnam' is dangerous, better use
>>> `mkstemp'
>>> > >> x86_64-diags-linux-gcc -m64 -march=haswell
>>> -mtune=haswell
>>> > -mfpmath=sse -mavx2
>>> > >>
>>> --sysroot=/projects/yocto-git/build/tmp/sysroots/continental -Wl,-O1
>>> > >> -Wl,--hash-style=gnu -Wl,--as-needed -Xlinker
>>> -export-dynamic -o
>>> > python \
>>> > >> Modules/python.o \
>>> > >> -L. -lpython2.7 -lpthread -ldl
>>> -lpthread
>>> > -lutil -lm
>>> > >>
>>> >
>>> _PYTHON_PROJECT_BASE=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build
>>> > >> _PYTHON_HOST_PLATFORM=linux2-x86_64
>>> > >>
>>> >
>>> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib/plat-linux2
>>> > >> python2.7 -S -m sysconfig --generate-posix-vars ;\
>>> > >> if test $? -ne 0 ; then \
>>> > >> echo "generate-posix-vars failed" ; \
>>> > >> rm -f ./pybuilddir.txt ; \
>>> > >> exit 1 ; \
>>> > >> fi
>>> > >> Illegal instruction (core dumped)
>>> > >> make: *** [sharedmods] Error 132
>>> > >> WARNING: exit code 1 from a shell command.
>>> > >> ERROR: oe_runmake failed
>>> > >> ERROR: Function failed: do_install (log file is located
>>> at
>>> > >>
>>> >
>>> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/temp/log.do_install.17258)
>>> > >>
>>> > >> Here is my tune-haswell.inc (ignore some copy/paste
>>> comment
>>> > issues right now =):
>>> > >>
>>> > >> # Settings for the GCC(1) cpu-type "haswell":
>>> > >> #
>>> > >> # Intel Core i7 CPU with 64-bit extensions, MOVBE,
>>> MMX, SSE,
>>> > SSE2, SSE3,·
>>> > >> # SSSE3, SSE4.1, SSE4.2, POPCNT, AVX, AVX2, AES,
>>> PCLMUL,
>>> > FSGSBASE,
>>> > >> # RDRND, FMA, BMI, BMI2 and F16C instruction set
>>> support.
>>> > >> #
>>> > >> # This tune is recommended for Intel Nehalem and
>>> Silvermont (e.g.
>>> > Bay Trail) CPUs
>>> > >> # (and beyond).
>>> > >> #
>>> > >> DEFAULTTUNE ?= "haswell"
>>> > >>
>>> > >> # Pull in the previous tune in to pull in
>>> PACKAGE_EXTRA_ARCHS
>>> > >> require conf/machine/include/tune-corei7.inc
>>> > >>
>>> > >> # Extra tune features
>>> > >> TUNEVALID[haswell] = "Enable haswell specific processor
>>> > optimizations"
>>> > >> TUNE_CCARGS .= "${@bb.utils.contains("TUNE_FEATURES",
>>> "haswell", "
>>> > >> -march=haswell -mtune=haswell -mfpmath=sse -mavx2", "",
>>> d)}"
>>> > >>
>>> > >> # Extra tune selections
>>> > >> AVAILTUNES += "haswell"
>>> > >> TUNE_FEATURES_tune-haswell =
>>> "${TUNE_FEATURES_tune-x86-64} haswell"
>>> > >> BASE_LIB_tune-haswell = "lib"
>>> > >> TUNE_PKGARCH_tune-haswell = "haswell"
>>> > >> PACKAGE_EXTRA_ARCHS_tune-haswell =
>>> > "${PACKAGE_EXTRA_ARCHS_tune-corei7} haswell"
>>> > >>
>>> > >>
>>> > >>
>>> > >
>>> >
>>> > --
>>> > _______________________________________________
>>> > yocto mailing list
>>> > yocto at yoctoproject.org <mailto:yocto at yoctoproject.org>
>>> > https://lists.yoctoproject.org/listinfo/yocto
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20151130/ca705190/attachment.html>
More information about the yocto
mailing list