[yocto] Question about rebuilding RPM package index for updated RPMs when bitbake completes

Tue Apr 22 13:03:07 PDT 2014

On 22/04/2014 19:14, Mark Hatle wrote:
> On 4/22/14, 1:50 AM, Alex J Lennon wrote:
>>
>> On 22/04/2014 02:36, Mark Hatle wrote:
>>> On 4/20/14, 7:15 AM, Alex J Lennon wrote:
>>>> Hi,
>>>>
>>>> I'm trying to put in place a development workflow using the PR server,
>>>> RPM package feeds and smart update/install on a target.
>>>>
>>>> I see that when I modify and rebuild my recipe, foo, the PR server
>>>> increments its count within the RPM filename, but the RPM feed data
>>>> doesn't seem to update.
>>>>
>>>
>>> ...
>>>
>>>> So if I am understanding the above correctly, when I make a change
>>>> to a
>>>> recipe and build it, PR automatically
>>>> updates, the old RPM is removed and the new RPM added to the feed
>>>> directory. However the package index
>>>> for the feed is not updated.
>>>>
>>>> So if at that point I try to make use of the feed on a target I am
>>>> likely to find something is broken.
>>>
>>> The feed is normally indexed (createrepo) either when you manually
>>> run the
>>> package-index operation, or when you construct a filesystem.  Until
>>> you do
>>> that, the feed directories are transient.
>>>
>>
>> Is there a reason for this? Surely having a package index that is out
>> of sync
>> with the packages until a manual operation is performed isn't ideal?
>
> Yes, if you re-indexed the set of packages after each package, you
> will add multiple minutes to the construction of each package, and
> introduce locking into the system.  (You won't be able to generate
> multiple recipes in parallel due to this.)  (I don't consider it a
> package feed until all of the packages are built and the index is
> constructed.)
>

I wasn't suggesting that the index is built mid-run . Not knowing the
internals of the bitbake dependency chain, presumably the package index
would/could be rebuilt at the end of the build process and so would not
block parallelisation (which I agree would be a time-sink)

>>>> If that is true would it make sense to leave the old RPM in the feed
>>>> directory until package-index
>>>> is re-ran, or to run package-index automatically at the end of a build
>>>> when RPMs have changed?
>>>
>>> I -never- export the feed directories from the project directory.
>>> Instead, I
>>> copy the packages from the feed directory to where I share them, and
>>> then run
>>> the indexer against the external repository.  This preserves the older
>>> versions and also makes the new ones available.
>>>
>>> To run the indexer I have to configure and run it manually...
>>>
>>> PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH
>>>
>>>
>>> /home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo
>>>
>>> --update -q <path to feed>
>>>
>>> So for qemux86_64, I end up running the above three times.  all,
>>> x86_64 and
>>> qemux86_64.
>>>
>>> Then on the target I just do:
>>>
>>> smart update
>>> smart upgrade -y
>>>
>>
>> Thanks, that's useful
>>
>> However I still believe the core question stands, which is why the
>> package index
>> is out of sync, and why it's a good thing to have to run bitbake
>> package-index
>
> The other option is to remove the repo indexes when packages are
> written. Again, I don't consider these out of sync though, as the
> files have different purposes and constraints.  They are simply
> constructed to enable the rootfs generation.  Upgrades are external of
> the build system.
>

Fair comment. I take the view that these feeds are available, could
easily be relied upon not to be transient, and I'd like to use them
locally to speed up my development workflow. I think other users would too.

Much of the value of the build environment is the automation. If bitbake
can do a task that means I don't have to remember to do it, and I don't
have to explain to somebody else that they need to do it, then that
seems like a win to me (even if that is an option I can turn on).

>> manually to bring the package index back into sync with the packages?
>>
>> I too export the feed directories to a server for 3rd party
>> consumption via
>> compressed rsync over SSH. I don't want to expose more on the server
>> than a
>> simple SSH endpoint, and rsync seems a sensible way to minimise
>> copying times,
>> so either I need the package index to be correct prior to the rsync,
>> I need to
>> export, say, an NFS filesystem which I don't want to do, or I need some
>> createrepo specifics on what I would prefer to be a dumb web-server.
>
> For an rsync, I would either index on the server using a cron-job.. so
> when new packages appear they get added to the index, or have a local
> mirror of the directory including the updated index... you just want
> to make sure the index files are synced last somehow.
>

To me the server side customisation brings in in extra work that
shouldn't be needed in all cases, (although will undoubtely be needed in
some cases).

I would much prefer not to have to worry about it.

If the package feed could be relied upon to be in sync with the contents
of the tmp/deploy/rpm/* tree at the end of a bitbake build then the
problem simply goes away, potentially at the cost of a little lost time
during the build.

Understanding that other use-cases will be different from mine I have
been wondering today,

- Can I add something into local.conf that would add a dependency to
cause the package-index task to run at the end of the bitbake build
  (preferably only if the contents of the package directories have changed?

- Does the build environment have the concept of post-build triggers?
i.e. Understanding that I could script something myself, is there a
recommended
  method for defining a post-build trigger script for bitbake to run on
certain conditions e.g to do the the package reindex and feed upload in
this case?

Thanks,

Alex

> --Mark
>
>> Best,
>>
>> Alex
>>
>>> --Mark
>>>
>>>> Thanks,
>>>>
>>>> Alex
>>>>
>>>>
>>>>
>>>
>>
>> -- 
>>
>> Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>
>>
>> Alex J Lennon / Director
>> 1 Queensway, Liverpool L22 4RA
>>
>> mobile: +44 (0)7956 668178
>>
>> Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
>> <skype:alexjlennon?add>
>>
>> This e-mail message may contain confidential or legally privileged
>> information
>> and is intended only for the use of the intended recipient(s). Any
>> unauthorized
>> disclosure, dissemination, distribution, copying or the taking of any
>> action in
>> reliance on the information herein is prohibited. E-mails are not
>> secure and
>> cannot be guaranteed to be error free as they can be intercepted,
>> amended, or
>> contain viruses. Anyone who communicates with us by e-mail is deemed
>> to have
>> accepted these risks. Company Name is not responsible for errors or
>> omissions in
>> this message and denies any responsibility for any damage arising
>> from the use
>> of e-mail. Any opinion and other statement contained in this message
>> and any
>> attachment are solely those of the author and do not necessarily
>> represent those
>> of the company.
>>
>
>
>

-- 

Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>

Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA

mobile: +44 (0)7956 668178

Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
<skype:alexjlennon?add>

This e-mail message may contain confidential or legally privileged
information and is intended only for the use of the intended
recipient(s). Any unauthorized disclosure, dissemination, distribution,
copying or the taking of any action in reliance on the information
herein is prohibited. E-mails are not secure and cannot be guaranteed to
be error free as they can be intercepted, amended, or contain viruses.
Anyone who communicates with us by e-mail is deemed to have accepted
these risks. Company Name is not responsible for errors or omissions in
this message and denies any responsibility for any damage arising from
the use of e-mail. Any opinion and other statement contained in this
message and any attachment are solely those of the author and do not
necessarily represent those of the company.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20140422/5482dffb/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ddlogo-4.png
Type: image/png
Size: 3997 bytes
Desc: not available
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20140422/5482dffb/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: linkedin.png
Type: image/png
Size: 631 bytes
Desc: not available
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20140422/5482dffb/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: skype.png
Type: image/png
Size: 800 bytes
Desc: not available
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20140422/5482dffb/attachment-0002.png>