[yocto] RFC: Improving the developer workflow

[Alex J Lennon]

On 07/08/2014 14:05, Paul Eggleton wrote:
> Hi Alex,
>
> On Thursday 07 August 2014 11:13:02 Alex J Lennon wrote:
>> On 07/08/2014 10:10, Paul Eggleton wrote:
>> fwiw Upgrade solutions are something that is still a read need imho,  as
>> I think we discussed at one of the FOSDEMs.
>>
>> (The other real need being an on-board test framework, again imho, and
>> which I believe is ongoing)
> Indeed; I think we've made some pretty good progress here in that the Yocto 
> Project QA team is now using the automated runtime testing to do QA tests on 
> real hardware. Reporting and monitoring of ptest results is also being looked 
> at as well as integration with LAVA.
>  

Great news. I really want to look into this but as ever time is the
constraining factor.

>> Historically I, and I suspect others, have done full image updates of
>> the storage medium,  onboard flash or whatever but these images are
>> getting so big now that I am trying to  move away from that and into
>> using package feeds for updates to embedded targets.
> Personally with how fragile package management can end up being, I'm convinced 
> that full-image updates are the way to go for a lot of cases, but ideally with 
> some intelligence so that you only ship the changes (at a filesystem level 
> rather than a package or file level). This ensures that an upgraded image on 
> one device ends up exactly identical to any other device including a newly 
> deployed one. Of course it does assume that you have a read-only rootfs and 
> keep your configuration data / logs / other writeable data on a separate 
> partition or storage medium. However, beyond improvements to support for 
> having a read-only rootfs we haven't really achieved anything in terms of out-
> of-the-box support for this, mainly due to lack of resources.

Deltas. Yes I've seen binary deltas attempted over the years, with
varying degrees of success.

I can see how what you say could work at a file-system level if we could
separate out the
writeable data, yes. Not sure I've seen any tooling around this though?

Back in the day when I first started out with Arcom Embedded Linux in
the late '90's I had us
do something similar with a read only JFFS2 system partition and then a
separate app/data
partition. That seemed to work OK. Maybe I need to revisit that.

> However, whilst I haven't had a chance to look at it closely, there has been 
> some work on this within the community:
>
> http://sbabic.github.io/swupdate/swupdate.html
> https://github.com/sbabic/swupdate
> https://github.com/sbabic/meta-swupdate/

I'll take a look. Thanks.

>  
>> My initial experience has been that
>>
>> - as you mention it would be really helpful to have something "more"
>> around management  of package feed releases / targets.
>>
>> - some automation around deployment of package feeds to production
>> servers would help,   or at least some documentation on best practice.
> So the scope of my proposal is a little bit narrower, i.e. for the SDK; and 
> I'm suggesting that we mostly bypass the packaging system since it doesn't 
> really add much benefit and sometimes gets in the way when you're an 
> application developer in the middle of development and the level of churn is 
> high (as opposed to making incremental changes after the product's release).

Mmm. Yes I can understand that. Same here.

>> The other big issue I am seeing, which is mostly my own fault thus far,
>> is that I have sometimes  taken  the easy option of modifying the root
>> filesystem image in various ways within the image recipe (for example
>> changing  a Webmin configuration perhaps)
>>
>> However when I then come to upgrade a package in-situ, such as Webmin,
>> the changes  are  then overwritten.
>>
>> I think this is probably also an issue when upgrading packages that have
>> had local modifications made, and I wonder whether there's a solution to
>> this that I'm not aware of?
> We do have CONFFILES to point to configuration files that may be modified (and 
> thus should not just be overwritten on upgrade). There's not much logic in the 
> actual build system to deal with this, we just pass it to the package manager; 
> but it does work, and recipes that deploy configuration files (and bbappends, if 
> the configuration file is being added rather than changed from there) should set 
> CONFFILES so that the right thing happens on upgrade if you are using a 
> package manager on the target.
>
> A related issue is that for anything other than temporary changes it's often 
> not clear which recipe you need to change/append in order to provide your own 
> version of a particular config file. FYI I entered the following enhancement bug 
> some months ago to add a tool to help with that:
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6447

Interesting thanks. I don't recall seeing this in recipes. I might have
missed it or are not many
people using this features in their recipes? Of course the next issue is
not knowing what you
want to do with those conf files during an unattended upgrade onto an
embedded box.

>> I am aware of course that mainstream package management tools allow
>> diffing, upgrading,  ignoring and such but I am unsure as to how that is
>> supported under Yocto at present?
> There isn't really any support for this at the moment, no; I think we'd want 
> to try to do this kind of thing at the build system end though to avoid tying 
> ourselves to one particular package manager.
>  

Indeed.

Cheers!

Alex