[yocto] Cross-compiling python fails with new custom tuning

[Michael Habibi]

For what it's worth, this is the offending portion of Makefile.pre:452:

# Create build directory and generate the sysconfig build-time data there.
# pybuilddir.txt contains the name of the build dir and is used for
# sys.path fixup -- see Modules/getpath.c.
# Since this step runs before shared modules are built, try to avoid
bootstrap
# problems by creating a dummy pybuilddir.txt just to allow interpreter
# initialization to succeed.  It will be overwritten by generate-posix-vars
# or removed in case of failure.
pybuilddir.txt: $(BUILDPYTHON)
→ @echo "none" > ./pybuilddir.txt
→ $(RUNSHARED) $(PYTHON_FOR_BUILD) -S -m sysconfig --generate-posix-vars ;\
→ if test $$? -ne 0 ; then \
→ → echo "generate-posix-vars failed" ; \
→ → rm -f ./pybuilddir.txt ; \
→ → exit 1 ; \
→ fi

I don't know enough about the Python build to understand what it's trying
to do, but perhaps replacing PYTHON_FOR_BUILD with HOSTPYTHON may be
acceptable?

I'm surprised this seems to work for other people, since this should be
failing for anyone using Python on a target platform different from their
host.

On Wed, Nov 25, 2015 at 9:02 AM, Mark Hatle <mark.hatle at windriver.com>
wrote:

> On 11/24/15 3:23 PM, Mark Hatle wrote:
> > My guess is that there is a bug in the python integration where it's not
> > realizing the host and target are different systems, so it's trying to
> run a
> > target program on your host.  Your host isn't haswell, so... Illegal
> Instruction
> > -- and the system stops.
>
> Just an FYI, I hit the same problem today.  I suspect it is the host
> trying to
> run target software and I'm looking into it.
>
> --Mark
>
> > The alternative would be something is running QEMU to execute a target
> binary
> > and QEMU doesn't have instruction support -- but that doesn't look like
> the case
> > here.
> >
> > --Mark
> >
> > On 11/24/15 3:06 PM, Michael Habibi wrote:
> >> All,
> >>
> >> I added a new machine definition with tuning parameters for haswell
> >> microarchitectures (basically just duplicated corei7 but tuned it for
> haswell).
> >> This seems to work correctly, but failed when running do_install for
> python
> >> toward the end of the build process. I am running with the Yocto 2.0
> framework,
> >> with very minimal changes to create a new distribution and machine for
> our
> >> custom embedded device. Note I had this distro configuration working
> before, and
> >> the only difference is I added a new machine with this tuning.
> >>
> >> I believe the issue is because, as part of the do_install step for
> Python, it
> >> attempts to run python on the local host, despite being cross-compiled.
> However,
> >> it is tuned for a processor that my host machine doesn't run, so I get
> an
> >> instruction exception.
> >>
> >> Did I do something weird? I figure python would be configured for
> >> cross-compiling and therefore wouldn't try to run the target version on
> the
> >> host, even for sanity tests. Here is the output of the failure:
> >>
> >> $ tail -20
> >>
> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/temp/log.do_install.17258
> >>
> >> x86_64-diags-linux-ar rc libpython2.7.a Modules/threadmodule.o
> >>  Modules/signalmodule.o  Modules/posixmodule.o  Modules/errnomodule.o
> >>  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o
> >>  Modules/_weakref.o  Modules/zipimport.o  Modules/symtablemodule.o
> >>  Modules/md5module.o Modules/md5.o  Modules/xxsubtype.o
> >> x86_64-diags-linux-ranlib libpython2.7.a
> >> Modules/posixmodule.o: In function `posix_tmpnam':
> >>
> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Modules/posixmodule.c:7575:
> >> warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'
> >> Modules/posixmodule.o: In function `posix_tempnam':
> >>
> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Modules/posixmodule.c:7522:
> >> warning: the use of `tempnam' is dangerous, better use `mkstemp'
> >> x86_64-diags-linux-gcc  -m64 -march=haswell -mtune=haswell -mfpmath=sse
> -mavx2
> >> --sysroot=/projects/yocto-git/build/tmp/sysroots/continental -Wl,-O1
> >> -Wl,--hash-style=gnu -Wl,--as-needed -Xlinker -export-dynamic -o python
> \
> >>                         Modules/python.o \
> >>                         -L. -lpython2.7 -lpthread -ldl  -lpthread
> -lutil   -lm
> >>
> _PYTHON_PROJECT_BASE=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build
> >> _PYTHON_HOST_PLATFORM=linux2-x86_64
> >>
> PYTHONPATH=/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/build/build/lib.linux2-x86_64-2.7:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib:/projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/Python-2.7.9/Lib/plat-linux2
> >> python2.7 -S -m sysconfig --generate-posix-vars ;\
> >>         if test $? -ne 0 ; then \
> >>                 echo "generate-posix-vars failed" ; \
> >>                 rm -f ./pybuilddir.txt ; \
> >>                 exit 1 ; \
> >>         fi
> >> Illegal instruction (core dumped)
> >> make: *** [sharedmods] Error 132
> >> WARNING: exit code 1 from a shell command.
> >> ERROR: oe_runmake failed
> >> ERROR: Function failed: do_install (log file is located at
> >>
> /projects/yocto-git/build/tmp/work/haswell-diags-linux/python/2.7.9-r1/temp/log.do_install.17258)
> >>
> >> Here is my tune-haswell.inc (ignore some copy/paste comment issues
> right now =):
> >>
> >> # Settings for the GCC(1) cpu-type "haswell":
> >> #
> >> #     Intel Core i7 CPU with 64-bit extensions, MOVBE, MMX, SSE, SSE2,
> SSE3,·
> >> #     SSSE3, SSE4.1, SSE4.2, POPCNT, AVX, AVX2, AES, PCLMUL, FSGSBASE,
> >> #     RDRND, FMA, BMI, BMI2 and F16C instruction set support.
> >> #
> >> # This tune is recommended for Intel Nehalem and Silvermont (e.g. Bay
> Trail) CPUs
> >> # (and beyond).
> >> #
> >> DEFAULTTUNE ?= "haswell"
> >>
> >> # Pull in the previous tune in to pull in PACKAGE_EXTRA_ARCHS
> >> require conf/machine/include/tune-corei7.inc
> >>
> >> # Extra tune features
> >> TUNEVALID[haswell] = "Enable haswell specific processor optimizations"
> >> TUNE_CCARGS .= "${@bb.utils.contains("TUNE_FEATURES", "haswell", "
> >> -march=haswell -mtune=haswell -mfpmath=sse -mavx2", "", d)}"
> >>
> >> # Extra tune selections
> >> AVAILTUNES += "haswell"
> >> TUNE_FEATURES_tune-haswell = "${TUNE_FEATURES_tune-x86-64} haswell"
> >> BASE_LIB_tune-haswell = "lib"
> >> TUNE_PKGARCH_tune-haswell = "haswell"
> >> PACKAGE_EXTRA_ARCHS_tune-haswell = "${PACKAGE_EXTRA_ARCHS_tune-corei7}
> haswell"
> >>
> >>
> >>
> >
>
> --
> _______________________________________________
> yocto mailing list
> yocto at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20151125/4dedc86f/attachment.html>