[yocto] cannot re-use shared state cache between build hosts

[Paul Eggleton]

Hi Andrea,

On Saturday, 2 September 2017 3:04:00 AM NZST Andrea Galbusera wrote:
> To be honest, the current server was indeed setup with python's
> SimpleHTTPServer... As you suggest, I checked the verbose debug log and
> noticed what's happening behind the apparently happy "Checking sstate
> mirror object availability" step. After a first "SState: Successful fetch
> test for" that I see correctly served with 200 on the server side, tests
> for any other sstate object suddenly and systematically fail with logs like
> this:
> 
> DEBUG: SState: Attempting to fetch file://7d/sstate:libxml2:i586-
> poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> DEBUG: Searching for 7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:
> 7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz in paths:
>     /home/vagrant/koan/morty/build/sstate-cache
> DEBUG: Defaulting to /home/vagrant/koan/morty/build/sstate-cache/7d/sstate:
> libxml2:i586-poky-linux:
2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> for 7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23b
> db86ac7ab32_package_qa.tgz
> DEBUG: Testing URL file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:
> 7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> DEBUG: For url ['file', '', '7d/sstate:libxml2:i586-poky-
> linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz', '',
> '', OrderedDict()] comparing ['file', '', '.*', '', '', OrderedDict()] to
> ['http', '192.168.33.1:8000', '
> /sstate-cache/PATH', '', '', OrderedDict([('downloadfilename', 'PATH')])]
> DEBUG: For url file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:
> 7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz returning
> http://192.168.33.1:8000/sstate-cache/7d/sstate%3Alibxml2%3Ai586-poky-linux%
> 3A2.9.4%3Ar0%3Ai586%3A3%3A7da8fc
> 3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz;downloadfilename=7d/sstate:
> libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab
> 32_package_qa.tgz
> DEBUG: checkstatus: trying again
> DEBUG: checkstatus() urlopen failed: <urlopen error [Errno 9] Bad file
> descriptor>
> DEBUG: SState: Unsuccessful fetch test for file://7d/sstate:libxml2:i586-
> poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> 
> Nothing is reported server-side for any of these failures... As you
> recommend, I'll try to setup something more "decent" for the HTTP server
> and see if it helps.

Right, FYI we were using plain SimpleHTTPServer in oe-selftest, and we had to 
change it because it simply isn't reliable when you throw multiple concurrent 
requests at it as sstate does. There's a ThreadingMixIn you can inherit (see 
meta/lib/oeqa/utils/httpserver.py) but you might as well set up a proper 
server if you're doing more than just testing.

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre