[yocto] Storing Sstate in S3 success stories?

[Erik Hoogeveen]

Hi Timothy,

The S3 protocol is HTTP(S) based, the overhead per object is quite significant. This is not much of a problem for large files but the sstate_cache contains mostly lots of really small files. I think in this case you’re better of storing the cache on a secondary EBS volume that you can attach as a regular block device to the EC2 instance. You can swtich on deletion protection to make it survive EC2 termination.

Since EBS volumes are quite a bit more expensive than S3 buckets  you could make snapshots to transfer the state between build runs in stead then you can destroy the EBS volume when nothing is running.

Alls the documentation about EBS is here https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html

Cheers,
Erik
On 26 Feb 2019, 02:45 +0100, Timothy Froehlich <tfroehlich at archsys.io>, wrote:
I've been spending a bit too long this past week trying to build up a reproducable build infrastructure in AWS and I've got very little experience with cloud infrastucture and I'm wondering if I'm going in the wrong direction. I'm attempting to host my sstate_cache as a mirror in a private S3 bucket, and I believe I have everything configured properly, including exposing the bucket to http requests, since I can wget files that I've previously synced up to the bucket. However if I add in the SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM) and barely seems to get anything. The EC2 instance is in the same region as the S3 bucket, roles have been configured properly to allow access, etc.

I'm not looking for help debugging this, I just want to know whether I'm right that hosting my sstate in an S3 bucket should work. I've only been able to find one mention of it being done with no reproduction hints.

--
_______________________________________________
yocto mailing list
yocto at yoctoproject.org
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.yoctoproject.org%2Flistinfo%2Fyocto&data=02%7C01%7C%7Cdc66c2ab9da44272220b08d69b8c2262%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636867423506178146&sdata=TjwcnaFaZQpk9iuxL16%2FosjuqEH2S7aXB16JjBIDGko%3D&reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190226/5090ddfa/attachment.html>