[yocto] Storing Sstate in S3 success stories?

[Timothy Froehlich]

This doesn't seem to be an issue. I have multiple files with plus signs in
their names that made it back down to my local cache without requiring a
rebuild (including the whole Linux kernel)

On Tue, Feb 26, 2019 at 11:35 AM Brian Walsh <brian at walsh.ws> wrote:

> On Mon, Feb 25, 2019 at 8:46 PM Timothy Froehlich <tfroehlich at archsys.io>
> wrote:
> >
> > I've been spending a bit too long this past week trying to build up a
> reproducable build infrastructure in AWS and I've got very little
> experience with cloud infrastucture and I'm wondering if I'm going in the
> wrong direction. I'm attempting to host my sstate_cache as a mirror in a
> private S3 bucket, and I believe I have everything configured properly,
> including exposing the bucket to http requests, since I can wget files that
> I've previously synced up to the bucket. However if I add in the
> SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM)
> and barely seems to get anything. The EC2 instance is in the same region as
> the S3 bucket, roles have been configured properly to allow access, etc.
> >
> > I'm not looking for help debugging this, I just want to know whether I'm
> right that hosting my sstate in an S3 bucket should work. I've only been
> able to find one mention of it being done with no reproduction hints.
> >
>
> A lot of the files end up with plus signs in the name. This causes
> problems with retrieving files through http access with S3. S3
> translates all plus signs to spaces, even those in the file path. So
> if my-file_v1.0+g1241876 actually exists as named in S3 an http
> request for that file will trigger the server to look for
> "my-file_v1.0 g1241876"
>
> I ran into this problem trying to host an opkg repository in S3 for
> upgrading.
>
> It may mostly work for you but there will be many files that it will
> never be able to find in your S3 hosted sstate.
>
> Maybe this has been fixed by AWS. I noticed the problem a year or two ago.
>
>
> https://stackoverflow.com/questions/36734171/how-to-decide-if-the-filename-has-a-plus-sign-in-it#36758133
> https://news.ycombinator.com/item?id=15398804
>


-- 
Tim Froehlich
Embedded Linux Engineer
tfroehlich at archsys.io
215-218-8955
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20190226/995fb487/attachment.html>