USB Drives reports write completion before the data is actually written. How can this be disabled. The data is cached before written it seems. How can I turn this off?
This is not unique to USB drives, it’s normal behavior for all drive writes on Linux machines.
Thanks for the information. How can I disable this?
This post is for Manjaro Linux but should give you some useful hints:
Disabling that is an extremely bad idea. It can quickly result in data corruption if something is unplugged while a write is in progress. The alternative is forcing a wait for the write to finish before the user can do anything else with the machine.
IO from storage devices is ssllloooowwwww in relation to almost everything else the machine does so it is standard with all IO, whether read or write, to cache and allow the processor to do other things while waiting for IO to complete.
IO timing is part of the kernel and as such mostly cannot be modified.
If you are concerned about knowing when writes to a USB device are complete before you unplug it, there is (within the file manager) an unmount function that forces the cache to complete writing before telling the user it is safe to unplug the device. There also is an ‘eject’ command within the cli interface that does similarly. NOTHING can speed up the actual write time, only force a flush of the cache, then tell you when it is safe to unplug the device.
Many thanks for the detailed information. I,am using eject function at the moment, the problem is I don’t know how long to wait before it completes. If I know at least how long to wait before unplugging the device, it is really useful and would solve the issue. Alternatively if the OS reports write completed when actual writing has been completed this would also solve the issue.
Which ‘eject function’ are you using? All of them that I am aware of provide a positive indication when they are done.
I,am using the above gnome extension for ejecting usb. It says data is being written cannot eject. However I do not get a time frame to wait until successful ejection.
Once data is written the tool un mounts the drive automatically
\When you click on the eject icon in the left column in the gnome file manager it waits until the write is complete then the device disappears. If a write is in progress it tells you so with a popup and you have to override the wait in the popup or it will not eject before the write is completed.
I realize the ‘eject’ command on the cli may not give a warning, but it does not complete and give back a prompt until the eject is complete. Ergo, it is telling you that the write and eject is completed when the prompt returns.
I do not understand how much more notification is needed.
Why in the world would you consider disabling the cache to be of benefit there?
The tool is doing exactly what 99.999+% of users expect in that it is completing the write before it will perform the eject.
If you want to remove the device sooner, please pull it out immediately. !!!
Then suffer with the corrupted data caused by your impatience.
Patience is a virtue, especially when dealing with a computer that is busily writing data.
Hi jeff v many thanks for your input. I have been consistently trying to switch to linux. This is required for my workflow in my laptop. This would make my work easier.when you have large amounts of data to be transferred in and out of external drives and constantly perform the eject action. I can gauge a time of completion say may be 20 mins or an hour to complete the data transfer. The OS right now reports that the data has been copied when actual data is still being written from the cache. I,am seeing some solutions on using hdparm not sure if it applies to fedora as well.
I agree patience is a virtue, When you have a fixed time fram for work, it makes easy to schedule things in order.
Writes are seen by the OS as completed when the data has been fully written into cache. Ejects and dismounts however still see the device as active until the cache has been fully emptied. The size of the cache as well as the actual write speed makes a difference in the time required to empty the cache so that can vary by device.
There is another command
sync that empties the cache and if issued on the cli it will not return a prompt until the IO cache is empty and all writes are completed, so maybe you could do a script that would issue that command (as root or using sudo) then when it returns the prompt you know it is safe to disconnect the device.
However, the app you are using, as well as the ‘eject’ icon in the gnome file manager both seem to make use of sync to flush the cache so a script would be just another tool and possibly not as efficient and complete.
Writing to a usb device is almost 100% of the time slower than reading from even a slow HDD. The delay is most likely the slow write. In days past when using floppies we had to to make sure the writes were complete so we used
sync ; sync ; sync then waited for the prompt before removing the floppy.
That disables caching on the device; it does not disable caching (buffering) in system memory before the data is sent to the device.
I think Asfar’s point (which noone on this thread has thus far acknowledged, so I will be the first) is that eject/sync/gnome’s eject button GIVE NO FEEDBACK WHATSOEVER of how long you’re going to have to wait.
By the way: “Just wait until it’s done” and “patience is a virtue” is not a valid, mature, friendly answer."
The GNOME a file copy dialog box has a % progress indication and ETA. And when that progress box disappears, it most obviously communicates that the copy is done.
But the truth, is that his Fedora Workstation has lied to him, perhaps to look like it’s faster and better than it really is. The file wasn’t actually copied yet after all, and worse yet, there’s no indication of when it will be.
This is a problem. It’s a real problem in UX design. It’s a Norman Door. And it’s one we’ve collectively had our heads in the sand about for a long time because of this virtuous patience mentality.
I’ve ran into this time and time again and I guess I’m just used to it:
I dd an iso to a usb stick for example, and dd exits giving me a prompt back and by all obvious indications, I should be able to pull the usb stick right away and use it, but I’m smart enough not to trust dd and I run sync a few times and wait for it first. But did we ever stop to ask, Should I?
How can we expect an innocent new user (who probably expects their software to be truthfull) to know about this?
I would expect there to be some counter in /sys or /dev that shows the number of blocks cached, waiting to flush to disk. A script could monitor this number and predict when it will reach 0 and display this in the task bar using gnome’s “executor” extension.
This isn’t just “How long you need to wait before removing a usb drive.”
It’s “How long do you need to wait before allowing the power grid to blink” and I don’t think we can expect anyone to predict that.
So from a human perspective it’s easy to understand how the lie of disk caching is unfriendly and puts the user at risk without disclosing that fairly and clearly.
I would agree that disabling disk caching is probably not the best way to fix this. BUT it probably would result in the GUI file copy indicator truthfully indicating when a file was fully on the usb stick.
This feature request belongs more with Gnome than Fedora, but I’d ask that the GUI file copy screen be modified to understand when a file is truly flushed to disk and represent that in its interface. And I seriously would consider some sort of “your data is not safe/is safe” indicator by the clock, which when hovered over would show what is pending writes and how long to expect it to take.
In response to your “it should tell me when it is done” above.
I just checked what I remembered with using the gnome file manager to copy a large file onto the usb device.
While the write is in progress when I attempt to eject the device (again within the file manager) it gives a popup telling the user it is busy. When the write has completed clicking on the eject icon dismounts it and a notification at the top center of the screen tells the user it can be unplugged. Both these actions are similar to what a user sees in windows. The file manager does have a pie type indicator that shows the write progress so the user can see how fast it is moving the data, though many other interfaces do not.
While I understand your desire to have something tell you “The copy will be done in XX minutes” there has never been a large demand that I have ever seen for that, but there has been considerable demand for the way things are now.
AFAIK nothing allows software to interfere with completion of data write. At the same time, NOTHING will prevent an impatient user from unplugging a usb device while it is in use. Only a users self control can do that. Even with a progress meter the user still has to watch it, which distracts from other things they may be doing.
Yes, it would be nice to have visible in all our tools the same progress meter we see in the gnome file manager (nautilus), but you also have to remember that this is open source software written by volunteers and designed according to the wishes of the developer of each package.
Should you want something specific then either A) write it yourself, or B) communicate with the developer as to what you would like to see and maybe the developer can find the time and inclination to add your feature into his product.
There is a learning curve when encountering anything new and newbies will encounter bumps and curves when first beginning, that they have to learn how to navigate. If you want something that always works only one way then stick with apple or microsoft so you can do things the one and only way allowed. FOSS is a lot of developers with a lot of different ideas of what is right and different ways of doing the same thing. Newbies have to learn about that with linux and hopefully each one is able to assist in making things better for everyone.
I agree completely. Hope there will be a solution in the future. Thanks for pointing things out.
Oh. My bad. I forgot emulating Microsoft Windows ™ was an objective of Fedora GNU+Linux. That’s the 4th f right…
If you haven’t read “The design of Everyday Things” by Don Norman, I highly recommend it.
Click here first, and then wait an arbitrary amount of time with no indication of how long it will be.
“No.” -said every human, ever.
Communication to the user is not optional.
While I certainly agree that this sort of mechanism would be useful, it will take a lot of work to get there. The work would need to start in the kernel, providing a mechanism to report the amount of ‘dirty’ pages in memory for a specific filesystem. Also, as is usually the case, a progress bar based on this number (or a percentage indicator) will not be 100% reliable, and if this mechanism existed we’d end up seeing users complain that when they clicked the ‘eject’ button the ‘expected time’ started out as 5 minutes but then crept up and the total time required ended up being 10 minutes. There are so many variables involved in the actual rate-of-writing to the filesystem that this will be a significant challenge.
I did run into file corruptions for constantly used files when I disabled the write caching. So I turned it back on