[yocto] File synchronization on ubifs/xfs

[Sebastian Brand]

> An application like "sed" does a rename after write, but that's totally
> unrelated to your issue. Sed does that because it cannot write to the file
> it's also reading from.

It might be the reason behind the rename, but you can also argue that
it is expected that a program that edits a file either leaves it as it was
or changed, but not empty, if the system crashes. (But you can also argue that
a power fail is undefined behavior.)

> If someone were to patch sed to insert an "fsync"
> before the rename, people would get very very angry at that person for making
> their build servers horrendously slow and draining their laptop batteries or
> something to that effect.

On ext3 (with mode data=ordered) it would indeed make the systems horrendously
slow. But on other file systems the impact should be much more reasonable.

> In the sed case, the proper thing to do is to direct sed's output to another
> file and then take care of syncing and renaming yourself.

That would be the preferable solution. If sed was not already heavily used to
update configuration files, and if many developers didn't "know" that sed -i
can be used for power-fail safe file updates.


The reason why I though that it might be interesting for other people using
Yocto is that this is a problem for many file systems that are used in
embedded systems. And it becomes a problem as soon as there is someone
that thinks that "sed -i" is safe on their system, since it is safe on ext4.
And for me it was more reasonable to implement a busybox feature than the
alternative solutions.


Cheers,
Sebastian
________________________________________
From: Mike Looijmans <mike.looijmans at topic.nl>
Sent: Monday, May 13, 2019 1:39 PM
To: Sebastian Brand; yocto at yoctoproject.org
Subject: Re: [yocto] File synchronization on ubifs/xfs

An application like "sed" does a rename after write, but that's totally
unrelated to your issue. Sed does that because it cannot write to the file
it's also reading from. If someone were to patch sed to insert an "fsync"
before the rename, people would get very very angry at that person for making
their build servers horrendously slow and draining their laptop batteries or
something to that effect.

When they tell you to fix "the application", they mean YOUR application, not
all the programs in the world that happen to call rename.

In the sed case, the proper thing to do is to direct sed's output to another
file and then take care of syncing and renaming yourself.



On 15-04-19 14:31, Sebastian Brand wrote:
> Hi all,
>
> I stumbled into an file synchronization problem, and I am curious if there are more people using  Yocto and ubifs/xfs/similar that have an interest of "fsync before rename"?
>
> When trying to atomically update a file a common flow is "create copy, edit copy, rename copy->original", but this does for some file systems not guarantee that the file is not empty after a power fail. A more correct flow is "create copy, edit copy, fsync copy, rename copy->original", at least according to file system developers. This has been discussed a lot and the general status seems to be that many file system developers think that applications that does not fsync before rename are broken, and many application developers think that fsync is implied at rename and this needs to be fixed in the file systems. (I am (trying to) not picking a side, and just trying to solve my problems :) )
>
> Since some file systems common in embedded systems (UBIFS, XFS) does not implement automatic fsync before rename, this becomes a bit of a problem if applications in Yocto does not sync "correctly". One of these applications is sed (in-place edit), and I am currently working on a patch to add fsync before rename (as a feature). Busybox are not interested in the feature (as they are of the opinion that this should be solved in the file system), but I am curious if there is any interest in these kind of features in Yocto?
>
> Regards,
> Sebastian
>