[yocto] Safely cleaning 'downloads'

[Christopher Larson]

On Thu, Oct 1, 2015 at 9:49 AM, Gary Thomas <gary at mlbassoc.com> wrote:

> On 2015-10-01 10:38, Smith, Virgil wrote:
>
>> The following is roughly the procedure I follow and that works for me.
>> Maybe someone could chime in with how some of this should be trimmed based
>> on yocto/bitbake intent/design.
>> Even so I'd probably stick with this level of extremism because without a
>> known good backup of your downloads(sources) you may be incapable of
>> (tweaking and) rebuilding your products if anything happens to your build
>> server.
>>
>> The only reason I've seen that simply deleting the downloads folder
>> causes problems is that external servers/content go away, violate their git
>> history, or replace files with non-identical contents.
>>
>>
>> Warning: The following does not maintain PR server information, so
>> automatic upgrading of your own packages could break.  If you rely on this
>> work out how to extract that information (and back it up regularly).
>>
>> 1. rename/move your current downloads folder and create a new one.
>> 2. for all of your product build configurations empty out the following
>> folders
>> 2.1 cache
>> 2.2 state-cache
>> 2.3 tmp
>> 3. build (bitbake) all your product images with all appropriate
>> configuration variances
>> 4. run the following command to extract the unexpanded sources from
>> downloads
>> find -H downloads -maxdepth 1 \
>>       -not -type d   -and   -not -name "*.done" \
>>       -exec cp -L {} sources-tmp \;
>>
>> You now have everything you *currently* need for a sources mirror in the
>> sources-tmp folder.
>>
>> 5. move sources-tmp to wherever/whatever backs your SOURCE_MIRROR_URL.
>> 6. Check those contents into some form of revision control (even if that
>> is just a manual set of backup folders/media).
>>
>>
>> Yes this is costs time and space, you just have to decide how much your
>> images and how much being able to reproduce them (with or without 'small'
>> changes) is worth.
>>
>
> I'm already doing more or less this same sequence.  I use these commands to
> stage the downloaded files to my mirror (/work/misc/Poky/sources for
> historical reasons)
>   ( cd downloads;
>     find . -maxdepth 1 -type f | grep -v '.lock$' | grep -v '.done$'
> >/tmp/files.$$;
>     rsync -auv --files-from=/tmp/files.$$ . /work/misc/Poky/sources
>   )
>
> This works very well (I've been doing it for many years).  The issue I'm
> trying
> to work on now is that my script leaves 'downloads' possibly full of
> files, especially
> if there are new versions that have just been downloaded.  This is
> especially noticeable
> for the tarballs of GIT trees - there are a number that I need/use that
> are measured in
> gigabytes (e.g. the RaspberryPi board firmware is 4194568645 bytes as of
> 2015-07-20!)
> Once I've saved these to my mirror(s), I'd like to be able to purge them
> from the local
> download directory in my builds.  As mentioned, I've found that just
> wiping that in a
> build tree tends to break things quite badly.  Of course I can always
> start over with
> a new build tree, but that also defeats the purpose of incremental builds.


I'd think something like this would get the job done:

1. Do a build of all your supported machines and configurations with
BB_GENERATE_MIRROR_TARBALLS=1 to ensure you have current, not out of date
scm tarballs.

2. Set up builds of all your supported machines and configurations, using a
new DL_DIR, with PREMIRRORS pointing to the old DL_DIR.

3. Either clean up the old DL_DIR by access time before you kicked off the
builds, or resolve the symlinks in the new DL_DIR and remove the old.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20151001/e44336b2/attachment.html>