[yocto] do_fetch hanging - any way to see debugging?

[Paul Eggleton]

Hi Michael,

On Wed, 02 Dec 2015 09:41:41 Michael Habibi wrote:
> I'm struggling with some network proxy issues and pulling down linux-yocto.
> I left my build running overnight and it simply hung at do_fetch. I tried
> using -v and -DDD but it doesn't actually show any output from the do_fetch
> recipe. I looked at the do_fetch logs and it doesn't have anything useful.
> Here are the last lines:
> 
> DEBUG: Fetcher failure: Fetch command failed with exit code 8, output:
> http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.g
> it.linux-yocto-4.1.git.tar.gz
> 
> 2015-12-02 09:21:15 ERROR 404: Not Found.
> 
> DEBUG: Trying Upstream
> DEBUG: Fetcher accessed the network with the command git -c
> core.fsyncobjectfiles=0 clone --bare --mirror
> http://git.yoctoproject.org/git/linux-yocto-4.1.git
> /projects/yocto-git/build/downloads/git2/git.yoctoproject.org.git.linux-yoct
> o-4.1.git DEBUG: Running export
> PATH="/projects/yocto-git/scripts:/projects/yocto-git/build/tmp/sysroots/x86
> _64-linux/usr/bin/x86_64-diags-linux:/projects/yocto-git/build/tmp/sysroots/
> continental/usr/bin/crossscripts:/projects/yocto-git/build/tmp/sysroots/x86_
> 64-linux/usr/sbin:/projects/yocto-git/build/tmp/sysroots/x86_64-linux/usr/bi
> n:/projects/yocto-git/build/tmp/sysroots/x86_64-linux/sbin:/projects/yocto-g
> it/build/tmp/sysroots/x86_64-linux/bin:/projects/yocto-git/scripts:/projects
> /yocto-git/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sb
> in:/bin:/usr/games:/usr/local/games:/home/habibim/bin"; export
> HOME="/home/habibim"; git -c core.fsyncobjectfiles=0 clone --bare --mirror
> http://git.yoctoproject.org/git/linux-yocto-4.1.git
> /projects/yocto-git/build/downloads/git2/git.yoctoproject.org.git.linux-yoct
> o-4.1.git
> 
> I looked at a log where this succeeded, and it didn't have much more
> information:
> 
> DEBUG: Running export
> PATH="/projects/yocto-git/scripts:/projects/yocto-git/build/tmp/sysroots/x86
> _64-linux/usr/bin/x86_64-poky-linux:/projects/yocto-git/build/tmp/sysroots/g
> enericx86-64/usr/bin/crossscripts:/projects/yocto-git/build/tmp/sysroots/x86
> _64-linux/usr/sbin:/projects/yocto-git/build/tmp/sysroots/x86_64-linux/usr/b
> in:/projects/yocto-git/build/tmp/sysroots/x86_64-linux/sbin:/projects/yocto-
> git/build/tmp/sysroots/x86_64-linux/bin:/projects/yocto-git/scripts:/project
> s/yocto-git/bitbake/bin:/projects/my-distro/scripts:/projects/my-distro/bitb
> ake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/ga
> mes:/usr/local/games"; export HOME="/home/habibim"; git -c
> core.fsyncobjectfiles=0 branch --contains
> aed902160251d69cc28d1e69a4f692e8ea8fa13b --list yocto-4.1 2> /dev/null | wc
> -l
> DEBUG: Python function base_do_fetch finished
> DEBUG: Python function do_fetch finished
> 
> Note that I am using a bbappend file for my custom machine:
> 
> KBRANCH_continental  = "standard/base"
> KMACHINE_continental ?= "common-pc-64"
> COMPATIBLE_MACHINE_continental = "continental"
> 
> # 4.1.13 rev tag
> LINUX_VERSION_continental = "4.1.13"
> SRCREV_machine_continental ?= "1f2ce4a2e7aea3a2123b17aff62a80553df31e21"
> 
> # Use http protocol
> SRC_URI_continental = "git://
> git.yoctoproject.org/git/linux-yocto-4.1.git;protocol=http;name=machine;bran
> ch=${KBRANCH} "
> 
> If I comment out my overrides for SRCREV and SRC_URI, it simply pulls the
> cache (somehow I managed to pull this down before). But if I use my
> overrides (to grab a newer kernel, and to use the http protocol), it will
> simply hang indefinitely. Any way I can debug what's happening?

If it's hung here it may be because the proxy you are behind is trying to 
download and scan the file in its entirety before it allows it to be passed 
through to your download process - we've seen this before. If that's what's 
happening there's not much we can do about that beyond increasing the timeout, 
because such proxies usually do not send anything back to the client in the 
way of status. If you need to you can set FETCHCMD_wget such that the -T 
option specifies a longer timeout value.

FWIW you should be able to see exactly which wget command line is being 
executed in the fetch log - you could try run this outside of the bitbake 
environment without the -nv switch to get more information (you'd need to 
ensure the proxy environment variables are set though or you wouldn't be 
replicating the same conditions).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre