[yocto] Cross-compiling python fails with new custom tuning

Michael Habibi mikehabibi at gmail.com
Wed Nov 25 08:01:26 PST 2015 

Ah sorry, I misspoke. I walked through the Makefile and configure scripts a
bit more, and $(PYTHON_FOR_BUILD) should in fact be a host version of
Python. It seems it's not being configured correctly. Either the path is
setup wrong and python2.7 is not pointing to the right version, or
python2.7 needs to be pointing to the absolute path instead of relying on
$PATH.




On Wed, Nov 25, 2015 at 9:45 AM, Michael Habibi <mikehabibi at gmail.com>
wrote:

> For what it's worth, this is the offending portion of Makefile.pre:452:
>
> # Create build directory and generate the sysconfig build-time data there.
> # pybuilddir.txt contains the name of the build dir and is used for
> # sys.path fixup -- see Modules/getpath.c.
> # Since this step runs before shared modules are built, try to avoid
> bootstrap
> # problems by creating a dummy pybuilddir.txt just to allow interpreter
> # initialization to succeed.  It will be overwritten by generate-posix-vars
> # or removed in case of failure.
> pybuilddir.txt: $(BUILDPYTHON)
> → @echo "none" > ./pybuilddir.txt
> → $(RUNSHARED) $(PYTHON_FOR_BUILD) -S -m sysconfig --generate-posix-vars ;\
> → if test $$? -ne 0 ; then \
> → → echo "generate-posix-vars failed" ; \
> → → rm -f ./pybuilddir.txt ; \
> → → exit 1 ; \
> → fi
>
> I don't know enough about the Python build to understand what it's trying
> to do, but perhaps replacing PYTHON_FOR_BUILD with HOSTPYTHON may be
> acceptable?
>
> I'm surprised this seems to work for other people, since this should be
> failing for anyone using Python on a target platform different from their
> host.
>
> On Wed, Nov 25, 2015 at 9:02 AM, Mark Hatle <mark.hatle at windriver.com>
> wrote:
>
>> On 11/24/15 3:23 PM, Mark Hatle wrote:
>> > My guess is that there is a bug in the python integration where it's not
>> > realizing the host and target are different systems, so it's trying to
>> run a
>> > target program on your host.  Your host isn't haswell, so... Illegal
>> Instruction
>> > -- and the system stops.
>>
>> Just an FYI, I hit the same problem today.  I suspect it is the host
>> trying to
>> run target software and I'm looking into it.
>>
>> --Mark
>>
>> > The alternative would be something is running QEMU to execute a target
>> binary
>> > and QEMU doesn't have instruction support -- but that doesn't look like
>> the case
>> > here.
>> >
>> > --Mark
>> >
>> > On 11/24/15 3:06 PM, Michael Habibi wrote:
>> >> All,
>> >>
>> >> I added a new machine definition with tuning parameters for haswell
>> >> microarchitectures (basically just duplicated corei7 but tuned it for
>> haswell).
>> >> This seems to work correctly, but failed when running do_install for
>> python
>> >> toward the end of the build process. I am running with the Yocto 2.0
>> framework,
>> >> with very minimal changes to create a new distribution and machine for
>> our
>> >> custom embedded device. Note I had this distro configuration working
>> before, and
>> >> the only difference is I added a new machine with this tuning.
>> >>
>> >> I believe the issue is because, as part of the do_install step for
>> Python, it
>> >> attempts to run python on the local host, despite being
>> cross-compiled. However,
>> >> it is tuned for a processor that my host machine doesn't run, so I get
>> an
>> >> instruction exception.
>> >>
>> >> Did I do something weird? I figure python would be configured for
>> >> cross-compiling and therefore wouldn't try to run the target version
>> on the
>> >> host, even for sanity tests. Here is the output of the failure:
>> >>
>> >> $ tail -20
>> >>
>> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/temp/log.do_install.17258
>> >>
>> >> x86_64-diags-linux-ar rc libpython2.7.a Modules/threadmodule.o
>> >>  Modules/signalmodule.o  Modules/posixmodule.o  Modules/errnomodule.o
>> >>  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o
>> >>  Modules/_weakref.o  Modules/zipimport.o  Modules/symtablemodule.o
>> >>  Modules/md5module.o Modules/md5.o  Modules/xxsubtype.o
>> >> x86_64-diags-linux-ranlib libpython2.7.a
>> >> Modules/posixmodule.o: In function `posix_tmpnam':
>> >>
>> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Modules/posixmodule.c:7575:
>> >> warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'
>> >> Modules/posixmodule.o: In function `posix_tempnam':
>> >>
>> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Modules/posixmodule.c:7522:
>> >> warning: the use of `tempnam' is dangerous, better use `mkstemp'
>> >> x86_64-diags-linux-gcc  -m64 -march=haswell -mtune=haswell
>> -mfpmath=sse -mavx2
>> >> --sysroot=/projects/yocto-git/build/tmp/sysroots/continental -Wl,-O1
>> >> -Wl,--hash-style=gnu -Wl,--as-needed -Xlinker -export-dynamic -o
>> python \
>> >>                         Modules/python.o \
>> >>                         -L. -lpython2.7 -lpthread -ldl  -lpthread
>> -lutil   -lm
>> >>
>> _PYTHON_PROJECT_BASE=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build
>> >> _PYTHON_HOST_PLATFORM=linux2-x86_64
>> >>
>> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib/plat-linux2
>> >> python2.7 -S -m sysconfig --generate-posix-vars ;\
>> >>         if test $? -ne 0 ; then \
>> >>                 echo "generate-posix-vars failed" ; \
>> >>                 rm -f ./pybuilddir.txt ; \
>> >>                 exit 1 ; \
>> >>         fi
>> >> Illegal instruction (core dumped)
>> >> make: *** [sharedmods] Error 132
>> >> WARNING: exit code 1 from a shell command.
>> >> ERROR: oe_runmake failed
>> >> ERROR: Function failed: do_install (log file is located at
>> >>
>> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/temp/log.do_install.17258)
>> >>
>> >> Here is my tune-haswell.inc (ignore some copy/paste comment issues
>> right now =):
>> >>
>> >> # Settings for the GCC(1) cpu-type "haswell":
>> >> #
>> >> #     Intel Core i7 CPU with 64-bit extensions, MOVBE, MMX, SSE, SSE2,
>> SSE3,·
>> >> #     SSSE3, SSE4.1, SSE4.2, POPCNT, AVX, AVX2, AES, PCLMUL, FSGSBASE,
>> >> #     RDRND, FMA, BMI, BMI2 and F16C instruction set support.
>> >> #
>> >> # This tune is recommended for Intel Nehalem and Silvermont (e.g. Bay
>> Trail) CPUs
>> >> # (and beyond).
>> >> #
>> >> DEFAULTTUNE ?= "haswell"
>> >>
>> >> # Pull in the previous tune in to pull in PACKAGE_EXTRA_ARCHS
>> >> require conf/machine/include/tune-corei7.inc
>> >>
>> >> # Extra tune features
>> >> TUNEVALID[haswell] = "Enable haswell specific processor optimizations"
>> >> TUNE_CCARGS .= "${@bb.utils.contains("TUNE_FEATURES", "haswell", "
>> >> -march=haswell -mtune=haswell -mfpmath=sse -mavx2", "", d)}"
>> >>
>> >> # Extra tune selections
>> >> AVAILTUNES += "haswell"
>> >> TUNE_FEATURES_tune-haswell = "${TUNE_FEATURES_tune-x86-64} haswell"
>> >> BASE_LIB_tune-haswell = "lib"
>> >> TUNE_PKGARCH_tune-haswell = "haswell"
>> >> PACKAGE_EXTRA_ARCHS_tune-haswell = "${PACKAGE_EXTRA_ARCHS_tune-corei7}
>> haswell"
>> >>
>> >>
>> >>
>> >
>>
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto at yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20151125/3431bc2e/attachment.html>

More information about the yocto mailing list