[yocto] Cross-compiling python fails with new custom tuning
Michael Habibi
mikehabibi at gmail.com
Mon Nov 30 16:29:36 PST 2015
Mark, great! I was literally about to try double-stacking the PATH next
(since removing both didn't work for reasons that became obvious after
running the experiment), so that it would grab the *.so's from
python-native and then the sysconfig from the python-cross. Looks like I
may need to subscribe to that list so I'm aware of these patches!
I'm not sure what the streams look like (I can't visualize it yet), but is
this change in any Yocto branch yet, or does it need to get pulled by
someone? I can merge it locally, but just curious how the OE-Core and Yocto
repos are related.
On Mon, Nov 30, 2015 at 6:07 PM, Mark Hatle <mark.hatle at windriver.com>
wrote:
> This is the patch that we ended up using:
>
>
> http://lists.openembedded.org/pipermail/openembedded-core/2015-November/113457.html
>
>
> http://lists.openembedded.org/pipermail/openembedded-core/2015-November/113456.html
>
>
> http://lists.openembedded.org/pipermail/openembedded-core/2015-November/113458.html
>
> On 11/30/15 5:12 PM, Michael Habibi wrote:
> > OK I've dug into this further today, and here are some findings:
> >
> > With my patch (which is similar to yours), it fails do_compile/package_qa
> > because of poisoned system directories during the compilation steps. This
> > approach seems to be a bust. Originally when I tested it, I hacked up the
> > Makefile and only re-ran do_install onward, and didn't re-run do_compile
> > (because I had to create the patch). Once I created the patch and re-ran
> from
> > clean, I got some errors.
> >
> > It appears the only reason it is using that directory in PYTHONPATH at
> all (the
> > one generated from the shell command and pybuilddir.txt) is for a
> potential
> > sysconfig file in there. I tracked it down to a comment in a bug on
> Python's bug
> > tracker:
> >
> > http://bugs.python.org/issue15484#msg180577
> >
> > If we change that directory, it will pull in the sysconfig settings from
> the
> > native Python configure (probably defaults to native sysconfig settings
> because
> > the PATH doesn't include the overrides), and then do some really bad
> things. The
> > question then became: in the unmodified build, why does do_install fail
> but
> > do_compile doesn't? They both seem to be using sharedmods recipe which
> uses
> > $PYTHONPATH. The answer is that when we run do_compile the first time
> around,
> > the offending *.so's are not generated yet, and thus are not included by
> > python-native. The second time around (in do_install), the *.so's are in
> the
> > path and so python-native will use them and override its own.
> >
> > My latest experiment is to remove that location altogether from
> PYTHONPATH and
> > see if it works (neither the shell-generated directory in python-cross,
> nor the
> > lib-dynload directory from python-native). The compile works the first
> time
> > through despite there being no sysconfig data, so I'm wondering if the
> install
> > step will also work without finding any data. python-native should
> continue to
> > work with its internal settings to find modules and libraries.
> >
> > To be continued tomorrow, I'm sure...
> >
> > On Mon, Nov 30, 2015 at 11:37 AM, Michael Habibi <mikehabibi at gmail.com
> > <mailto:mikehabibi at gmail.com>> wrote:
> >
> > Mark, I am continuing to look at this. I hope you don't mind if I
> keep
> > updating this thread with my investigation.
> >
> > First, I figured out my confusion. I was associating the wrong
> Makefile
> > command with the illegal instruction. It was actually the command
> under the
> > recipe for 'sharedmods', but it was silenced in the Makefile so I
> assumed it
> > was another line. You're right that the path with the *.so's is the
> > offending path. I've scoured online patches and can't understand how
> this
> > mechanism works outside of Yocto. I figure people trying to compile
> for ARM
> > would have similar issues. There seems to be a fundamental problem
> with how
> > Python sets up PYTHONPATH using the pybuilddir file and
> $abs_builddir.
> > However, other people seem to have successfully built python for
> other
> > architectures, so I'm still confused.
> >
> > I hardcoded a fix for PYTHONPATH to use the host lib-dynload
> directory, and
> > the python recipe successfully passed. Can you elaborate on the
> issues you
> > were having with your patch? The only difference is that I
> maintained the
> > other two directories in $PYTHONPATH,
> > e.g. $(srcdir)/Lib:$(srcdir)/Lib/plat-$(MACHDEP), and only replaced
> the
> > offending one.
> >
> > On Wed, Nov 25, 2015 at 4:58 PM, Michael Habibi <
> mikehabibi at gmail.com
> > <mailto:mikehabibi at gmail.com>> wrote:
> >
> > Mark,
> >
> > I ran the same command, including the offending PYTHONPATH, from
> the
> > shell without error. Any reason why that would work, despite it
> not
> > working from within the Makefile? I was thinking it was another
> > environment variable that was causing the issue (one that's not
> set in
> > my bash environment by default). My $PATH didn't have
> python-native so I
> > called it using an absolute path, but otherwise it's the same
> command
> > run from the Makefile (parsed, of course):
> >
> > bash$
> >
> _PYTHON_PROJECT_BASE=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build
> > _PYTHON_HOST_PLATFORM=linux2-x86_64
> >
> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib/plat-linux2
> >
> /projects/yocto-git/build/tmp/sysroots/x86_64-linux/usr/bin/python-native/python2.7
> > -S -m sysconfig --generate-posix-vars
> > bash$ ls pybuilddir.txt
> > pybuilddir.txt
> >
> > Maybe I'm missing something.
> >
> >
> > On Wed, Nov 25, 2015 at 4:38 PM, Mark Hatle <
> mark.hatle at windriver.com
> > <mailto:mark.hatle at windriver.com>> wrote:
> >
> > On 11/25/15 10:11 AM, Michael Habibi wrote:
> > > Well, I'm wrong yet again. A 'which python2.7' shows that
> it's pointing to a
> > > native version of Python:
> > >
> > > *| which python2.7*
> > > *|
> > >
> >
> /projects/yocto-git/build/tmp/sysroots/x86_64-linux/usr/bin/python-native/python2.7*
> > > |
> > >
> _PYTHON_PROJECT_BASE=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build
> > > _PYTHON_HOST_PLATFORM=linux2-x86_64
> > >
> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib/plat-linux2
> > > python2.7 -S -m sysconfig --generate-posix-vars ;\
> > > | if test $? -ne 0 ; then \
> > > | echo "generate-posix-vars failed" ; \
> > > | rm -f ./pybuilddir.txt ; \
> > > | exit 1 ; \
> > > | fi
> > > | Illegal instruction (core dumped)
> > > | make: *** [sharedmods] Error 132
> > >
> > > I guess one of the environmental variables being passed in
> is screwing up what
> > > it's doing. Unfortunately I don't know enough about the
> inner workings of Python
> > > to be of much help moving forward. I gave it my best shot!
> >
> > The problem I tracked down was:
> >
> >
> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7
> >
> > This is full of .so objects when loaded cause the illegal
> instruction.
> >
> > I came up with a patch that I thought was going to fix it,
> but it's
> > triggered
> > other failures.
> >
> >
> http://lists.openembedded.org/pipermail/openembedded-core/2015-November/113341.html
> >
> > Just to be clear.. this does NOT work properly in many
> cases. But
> > might give
> > you or someone else a clue as to how to possibly fix it.
> >
> > --Mark
> >
> > >
> > > On Wed, Nov 25, 2015 at 10:01 AM, Michael Habibi <
> mikehabibi at gmail.com <mailto:mikehabibi at gmail.com>
> > > <mailto:mikehabibi at gmail.com <mailto:mikehabibi at gmail.com>>>
> wrote:
> > >
> > > Ah sorry, I misspoke. I walked through the Makefile
> and configure scripts a
> > > bit more, and $(PYTHON_FOR_BUILD) should in fact be a
> host version of
> > > Python. It seems it's not being configured correctly.
> Either the path is
> > > setup wrong and python2.7 is not pointing to the right
> version, or python2.7
> > > needs to be pointing to the absolute path instead of
> relying on $PATH.
> > >
> > >
> > >
> > >
> > > On Wed, Nov 25, 2015 at 9:45 AM, Michael Habibi <
> mikehabibi at gmail.com <mailto:mikehabibi at gmail.com>
> > > <mailto:mikehabibi at gmail.com <mailto:
> mikehabibi at gmail.com>>>
> > wrote:
> > >
> > > For what it's worth, this is the offending portion
> of
> > Makefile.pre:452:
> > >
> > > # Create build directory and generate the sysconfig
> > build-time data there.
> > > # pybuilddir.txt contains the name of the build
> dir and is
> > used for
> > > # sys.path fixup -- see Modules/getpath.c.
> > > # Since this step runs before shared modules are
> built,
> > try to avoid
> > > bootstrap
> > > # problems by creating a dummy pybuilddir.txt just
> to
> > allow interpreter
> > > # initialization to succeed. It will be
> overwritten by
> > generate-posix-vars
> > > # or removed in case of failure.
> > > pybuilddir.txt: $(BUILDPYTHON)
> > > → @echo "none" > ./pybuilddir.txt
> > > → $(RUNSHARED) $(PYTHON_FOR_BUILD) -S -m sysconfig
> > --generate-posix-vars ;\
> > > → if test $$? -ne 0 ; then \
> > > → → echo "generate-posix-vars failed" ; \
> > > → → rm -f ./pybuilddir.txt ; \
> > > → → exit 1 ; \
> > > → fi
> > >
> > > I don't know enough about the Python build to
> understand
> > what it's
> > > trying to do, but perhaps replacing
> PYTHON_FOR_BUILD with
> > HOSTPYTHON may
> > > be acceptable?
> > >
> > > I'm surprised this seems to work for other people,
> since
> > this should be
> > > failing for anyone using Python on a target
> platform
> > different from
> > > their host.
> > >
> > > On Wed, Nov 25, 2015 at 9:02 AM, Mark Hatle
> > <mark.hatle at windriver.com <mailto:mark.hatle at windriver.com>
> > > <mailto:mark.hatle at windriver.com
> > <mailto:mark.hatle at windriver.com>>> wrote:
> > >
> > > On 11/24/15 3:23 PM, Mark Hatle wrote:
> > > > My guess is that there is a bug in the python
> > integration where
> > > it's not
> > > > realizing the host and target are different
> systems,
> > so it's
> > > trying to run a
> > > > target program on your host. Your host isn't
> > haswell, so...
> > > Illegal Instruction
> > > > -- and the system stops.
> > >
> > > Just an FYI, I hit the same problem today. I
> suspect
> > it is the host
> > > trying to
> > > run target software and I'm looking into it.
> > >
> > > --Mark
> > >
> > > > The alternative would be something is
> running QEMU
> > to execute a
> > > target binary
> > > > and QEMU doesn't have instruction support --
> but
> > that doesn't look
> > > like the case
> > > > here.
> > > >
> > > > --Mark
> > > >
> > > > On 11/24/15 3:06 PM, Michael Habibi wrote:
> > > >> All,
> > > >>
> > > >> I added a new machine definition with tuning
> > parameters for haswell
> > > >> microarchitectures (basically just
> duplicated
> > corei7 but tuned it
> > > for haswell).
> > > >> This seems to work correctly, but failed
> when
> > running do_install
> > > for python
> > > >> toward the end of the build process. I am
> running
> > with the Yocto
> > > 2.0 framework,
> > > >> with very minimal changes to create a new
> > distribution and
> > > machine for our
> > > >> custom embedded device. Note I had this
> distro
> > configuration
> > > working before, and
> > > >> the only difference is I added a new
> machine with
> > this tuning.
> > > >>
> > > >> I believe the issue is because, as part of
> the
> > do_install step
> > > for Python, it
> > > >> attempts to run python on the local host,
> despite being
> > > cross-compiled. However,
> > > >> it is tuned for a processor that my host
> machine
> > doesn't run, so
> > > I get an
> > > >> instruction exception.
> > > >>
> > > >> Did I do something weird? I figure python
> would be
> > configured for
> > > >> cross-compiling and therefore wouldn't try
> to run
> > the target
> > > version on the
> > > >> host, even for sanity tests. Here is the
> output of
> > the failure:
> > > >>
> > > >> $ tail -20
> > > >>
> > >
> >
> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/temp/log.do_install.17258
> > > >>
> > > >> x86_64-diags-linux-ar rc libpython2.7.a
> > Modules/threadmodule.o
> > > >> Modules/signalmodule.o
> Modules/posixmodule.o
> > Modules/errnomodule.o
> > > >> Modules/pwdmodule.o Modules/_sre.o
> > Modules/_codecsmodule.o
> > > >> Modules/_weakref.o Modules/zipimport.o
> > Modules/symtablemodule.o
> > > >> Modules/md5module.o Modules/md5.o
> Modules/xxsubtype.o
> > > >> x86_64-diags-linux-ranlib libpython2.7.a
> > > >> Modules/posixmodule.o: In function
> `posix_tmpnam':
> > > >>
> > >
> >
> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Modules/posixmodule.c:7575:
> > > >> warning: the use of `tmpnam_r' is
> dangerous, better
> > use `mkstemp'
> > > >> Modules/posixmodule.o: In function
> `posix_tempnam':
> > > >>
> > >
> >
> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Modules/posixmodule.c:7522:
> > > >> warning: the use of `tempnam' is dangerous,
> better
> > use `mkstemp'
> > > >> x86_64-diags-linux-gcc -m64 -march=haswell
> > -mtune=haswell
> > > -mfpmath=sse -mavx2
> > > >>
> > --sysroot=/projects/yocto-git/build/tmp/sysroots/continental
> -Wl,-O1
> > > >> -Wl,--hash-style=gnu -Wl,--as-needed
> -Xlinker
> > -export-dynamic -o
> > > python \
> > > >> Modules/python.o \
> > > >> -L. -lpython2.7
> -lpthread
> > -ldl -lpthread
> > > -lutil -lm
> > > >>
> > >
> >
> _PYTHON_PROJECT_BASE=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build
> > > >> _PYTHON_HOST_PLATFORM=linux2-x86_64
> > > >>
> > >
> >
> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib/plat-linux2
> > > >> python2.7 -S -m sysconfig
> --generate-posix-vars ;\
> > > >> if test $? -ne 0 ; then \
> > > >> echo "generate-posix-vars
> failed" ; \
> > > >> rm -f ./pybuilddir.txt ; \
> > > >> exit 1 ; \
> > > >> fi
> > > >> Illegal instruction (core dumped)
> > > >> make: *** [sharedmods] Error 132
> > > >> WARNING: exit code 1 from a shell command.
> > > >> ERROR: oe_runmake failed
> > > >> ERROR: Function failed: do_install (log
> file is
> > located at
> > > >>
> > >
> >
> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/temp/log.do_install.17258)
> > > >>
> > > >> Here is my tune-haswell.inc (ignore some
> copy/paste
> > comment
> > > issues right now =):
> > > >>
> > > >> # Settings for the GCC(1) cpu-type
> "haswell":
> > > >> #
> > > >> # Intel Core i7 CPU with 64-bit
> extensions,
> > MOVBE, MMX, SSE,
> > > SSE2, SSE3,·
> > > >> # SSSE3, SSE4.1, SSE4.2, POPCNT, AVX,
> AVX2,
> > AES, PCLMUL,
> > > FSGSBASE,
> > > >> # RDRND, FMA, BMI, BMI2 and F16C
> instruction
> > set support.
> > > >> #
> > > >> # This tune is recommended for Intel
> Nehalem and
> > Silvermont (e.g.
> > > Bay Trail) CPUs
> > > >> # (and beyond).
> > > >> #
> > > >> DEFAULTTUNE ?= "haswell"
> > > >>
> > > >> # Pull in the previous tune in to pull in
> > PACKAGE_EXTRA_ARCHS
> > > >> require conf/machine/include/tune-corei7.inc
> > > >>
> > > >> # Extra tune features
> > > >> TUNEVALID[haswell] = "Enable haswell
> specific processor
> > > optimizations"
> > > >> TUNE_CCARGS .=
> > "${@bb.utils.contains("TUNE_FEATURES", "haswell", "
> > > >> -march=haswell -mtune=haswell -mfpmath=sse
> -mavx2",
> > "", d)}"
> > > >>
> > > >> # Extra tune selections
> > > >> AVAILTUNES += "haswell"
> > > >> TUNE_FEATURES_tune-haswell =
> > "${TUNE_FEATURES_tune-x86-64} haswell"
> > > >> BASE_LIB_tune-haswell = "lib"
> > > >> TUNE_PKGARCH_tune-haswell = "haswell"
> > > >> PACKAGE_EXTRA_ARCHS_tune-haswell =
> > > "${PACKAGE_EXTRA_ARCHS_tune-corei7} haswell"
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > > --
> > > _______________________________________________
> > > yocto mailing list
> > > yocto at yoctoproject.org <mailto:
> yocto at yoctoproject.org>
> > <mailto:yocto at yoctoproject.org <mailto:
> yocto at yoctoproject.org>>
> > > https://lists.yoctoproject.org/listinfo/yocto
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20151130/d40b0541/attachment.html>
More information about the yocto
mailing list