[yocto] Fetcher error reporting

[Darren Hart]

On 10/22/2010 04:38 PM, Richard Purdie wrote:
> On Fri, 2010-10-22 at 11:22 -0700, Darren Hart wrote:
>> Here is an example of the do_fetch/do_unpack failure that I believe is
>> representative of what Dirk saw and what I have been seeing periodically.
>>
>> I saw the following while building libproxy in poky-image-sdk-live on a
>> Fedora 13 box behind the firewall. I saw identical failures with opkg on
>> a different machine - also behind the firewall. Both SRC_URIs point to
>> googlecode.
>>
>> log.do_fetch:
>> -------------
>> NOTE: fetch http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz
>> NOTE: libproxy-0.4.3:
>> http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz has no entry
>> in conf/checksums.ini, not checking URI
>>
>> log.do_unpack:
>> --------------
>> gzip: stdin: unexpected end of file
>> tar: Child returned status 1
>> tar: Exiting with failure status due to previous errors
>> NOTE: Unpacking
>> /vol/1/dvhart/poky.git/build/downloads/libproxy-0.4.3.tar.gz to
>> /vol/1/dvhart/poky.git/build/tmp/work/i586-poky-linux/libproxy-0.4.3-r2/
>> ERROR: Task failed:
>>
>> With opkg, I tried the following which resulted in the same error.
>>
>> $ bitbake -c clean;
>> $ rm downloads/*opkg*
>> $ bitbake opkg
>>
>> In both cases, using wget to manually download the file and place it in
>> the downloads directory allowed the build to continue.
>
> We've talked about this a lot over jabber. We don't have a reproducer.
> What we do see is its always the wget fetcher that hits this (for mirror
> urls in the case of git repos) and its failing creating a zero length
> file.
>
> If we make the assumption that wget is creating the empty file and
> exiting with an error code and look at what bitbake would do, it would
> give the behaviour described in these bugs.
>
> Why? If the original fetch fails, it falls back to the mirror code. That
> looks, sees a file on the disk and says "nothing to do".
>
> Solution is therefore to make sure if the wget fetcher fails we wipe any
> file that may be present.
>
> Is is possible the wget fetcher is creating a zero length file and no
> setting an error code but I really hope its not that broken.
>
> So I've pushed this patch as a bandaid for the problem:
>
> http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=c5fab99a6f979a4a0ce246c6395b35a3082aec0d
>

I've applied and am running another build. This looks sane to me. If I 
still run into this, I think the next step is to do a brute force 
zero-length test, and throw the FetchError and take this modified path 
from this patch in that case too.

wget will generated a 0 length file when using -O (which we confirmed we 
do not) and there are old bugs against it for creating 0 length files 
when a valid host fails to connect (without the -O option) but I wasn't 
able to reproduce that with a current wget.

Here's to hoping this fixes it.

-- 
Darren Hart
Embedded Linux Kernel