Backup slows down after installing F-33

For my backups I use my own program. On the first run it duplicates
the given directories to an identical structure on the given backup
media. On following runs it compares the previously written backup
structure with the current source and again duplicates the source to
a new identical structure on the given backup media. Only this time
it copies only files which were added or changed. The files that are
unchanged are hardlinked to the previous version, which saves both
time and space, sacrificing redundancy. The program is written in Perl.

I have been using this program since 2013. The backups are nominally
50 GB, but the daily differences are vastly smaller, so that the daily
backup takes about 3 to 4 minutes. The backups use ext2 filesystem.

After I upgraded from fc32 to fc33, the times for the backup have
suddenly risen by a factor betwen 4 and 5. It seems that it is the
writing / linking that takes the additional time. Reading the directory
data doesn’t seem to matter. I did a direct dnf upgrade, so that AFAIK
my FS should still be ext4.

Any ideas? Is fc33 doing some strange transformations from ext4 to
Btrfs and back? Is Perl under fc33 slower? …?

First, I am still gms, but as I discovered, my original email account is gone, which
probably caused my Fedora gms account to be blocked, so here I am as gms2.

It turns out, that during the backup I get into swapping mode. Either the OS now needs
more memory, or Perl doesn’t do it’s memory management as well as before, but
now I have at least a direction to move on. I would delete the above posting, but as
my gms account is unreachable (to me)…

1 Like

Would you be willing to post your program so we can test what happens on a different machine.

Also, info on your hardware would be nice.

 sudo inxi -F 

Please post the output in code tags [code] … [/code]

Is fc33 doing some strange transformations from ext4 to Btrfs and back?

No.

Regression testing is a bit of a pain, because you need an A and B setup to run the tests. And then it’s a case of process of elimination. bcc-tools have quite a lot of tools for finding latency. Of course you need to have some idea or reference for expected latency. But what you’re experiencing sounds like the latency is so high that it should be obvious if only you can narrow down where it’s happening. So bcc-tools can show you biolatency for the physical drive (not likely, it hasn’t changed), various file system latency tools (also no change, not likely), and a VFS latency tool (maybe). But there’s dozens of these and one might help narrow down what has such high latency.

Another possibility is to run something like perf top -p on the PID doing all the work and see what it’s doing and if it’s doing something unexpected. There’s a bunch of guides about perf tools.

I would prefer not to post it - it is 1300+ lines of ugly Perl code plus
some C-code and it needs some additional setup. Also in the meantime
I have discovered that a) the swap is probably just a red herring and
b) it sometimes looks like even a normal “cp -a” is slower. Of course,
since I didn’t do any measurements before the upgrade, the objective
value of that statement is around zero.
BTW inxi doesn’t come default, but looks interesting. Thank you for
the pointer.

Tell me about PITA… I don’t really want to start poking and prodding, I have
“better things to do”. What I really wanted to know was, if there was some
quantum jump between fc32 and fc33. “It just was normal”, then I did the
upgrade and “the performance went to hell”. I am using that code on 4 other
machines, but only one has the data size which makes it noticeable. I guess,
I will just figure some way around it (split into slow and fast changing data…).

This is a guess, but the swap-on-zram feature is applied on upgrades. Perhaps there’s some edge case.

Try this:

touch /etc/systemd/zram-generator.conf

And reboot, then try to reproduce the problem. If it reproduces it’s something else and have no idea what it could be.

Don’t delete it, even if you recover your gms account - the post may be a helpful lesson to others. Instead select a specific post as Solution to point readers into the right direction.

Thank you, that is exactly the type of answer I was looking for.
But - sorry - it didn’t help.

Once I get there… :-/

I suggest removing that file then. You’re probably better off with swap on zram than not. As for what else it could be, no idea.