Best practise updating kernel and systemd

Hello,

so far, I run “dnf update” as soon as I see the dragora notifier and then reboot, because I believe the kernel and systemd require it. But these reboots happen too often. What If I don’t reboot after kernel and systemd updates? Let’s say the new code is not yet in use and another update arrives on top of it? What about the auto removal of older kernels?

Thanks for any hints and pointers,
P.

How often are you getting kernel and systemd updates. They aren’t that common…

The kernel is typically loaded into memory. Even if the files on the disk change, the kernel should keep running. One source of potential issues comes in when you need a module that isn’t already loaded and the modules are no longer available on disk. Of course, if you are keeping 3 versions of kernels then you really aren’t rebooting enough if you have that problem. Even if you aren’t, it probably isn’t often that you would need modules that aren’t already loaded.


I am not familiar enough with systemd to know the impact there. I usually just reboot when there is a systemd/kernel update. Hopefully someone else will chime in.

In my personal experience, the most common problems comes from userspace issues where complex software like DEs will start having issues when library versions no longer match when something is loaded into memory tries to work with a library that now has a different version on disk.


All that being said, since the question is about best practice, I think it is clear that the best practice is to reboot when you do updates to critical components.

1 Like

Hi,

I tend to only reboot for kernel updates, which on fedora occurs about once a week. I only restart for systemd if there is security issue or a bug being patched, occurrences of fixes being required isn’t very often. So on average I only need to reboot for an update, about once a week.

If you still wish to restart systemd, you can do so with out a reboot:

sudo systemctl daemon-reexec

More details can be found here:

https://www.cyberciti.biz/faq/how-to-restart-systemd-without-rebooting-linux-when-critical-libraries-installed/

Thanks Tom.

to the 1. Q.:
I would like to say it depends if these updates fixes serious bugs and if these serious bugs concerns (correct english ?) to your box, e.g. kernel fixed wifi stuff, but you don’t run wifi, etc. …

I usually check what is fixed (example for kernels)
here:
https://www.kernel.org/
click “changelog” !
be aware: always check the correct version (here: 5.13.7) !

other sources:
Fedora’s package update DB
search for the component (e.g. kernel, systemd)
click it and then click the tab “Builds”, click on the package and then scroll down to “Changelog”

for the systemd case:
If you’re doing updates on the command line via “sudo dnf update” sometimes you see (after scrolling back) if a service needs restarting via “sudo systemctl daemon-reload” (with F35 -IIRC- it’s done automatically)

that said:
a restart of your box depends on your needs.
I usually don’t restart my box after updates except when a kernel update fixes a serious bug e.g. running the old kernel could lead to an damaged filesystem or has security holes or such.

usually serious bugs are communicated on mailling lists, etc.

to the 2. Q.:
I’ve never seen such a so short following update e.g. running “dnf update” brought a new kernel and running a second “dnf update” brings another new kernel.

apart from kernel code ([up to now] kernels need a reboot) I’m currently unsure if the updated code of a system sub-component, e.g. a library of Xorg, is immediately “sharp” without a restart of something.

to the 3. Q.:
-AFAIK- it’s given in /etc/dnf/dnf.conf and there “installonly_limit=3” says only 3 kernels on the box.
The installation of a new kernel triggers the removal of the oldest kernel (but only in the case 3 kernels are already on the box)

1 Like

Sorry for being late, Topic already is resolved.

There’s a tool called “needs-restarting”.
It is included in a package named “dnf-utils”.
You can check if a reboot is required or not and why.

So to install it:
sudo dnf install dnf-utils

To see if a reboot might be required:
needs-restarting -r

It can be used in scripts as well, since it will return 0 or 1. (the return code you can get witch echo $?)

man needs-restarting


EDIT


As pointed out by dalto installing “dnf-utils” is not needed any more.
dnf needs-restarting -r
It is included in dnf and does the same.

7 Likes

it’s never too late (even with a info I, as a longtime Fedora user, haven’t had yet)
:wink:

1 Like

I think if your solution is useful, the user still can change the solution. You have 28 days to be late before the topic get closed :smiley:

2 Likes

There isn’t really a need to install dnf-utils for that, is there?

If you look at it, all it does is calls dnf needs-restarting

4 Likes

Ah, I see. This wasn’t included in yum, so one needed to install yum-utils.
I assumed that this would be the same with dnf.
But in dnf it is built in. Cool thing. Thank you, didn’t know that.

1 Like

You may need to reboot Fedora after an update. You only need to reboot if files are in use or a new kernel is installed. If files are in use or a new kernel is installed then a reboot is required. (Files would be in use like an updated libcurl or libcrypto or libssl).

To check if a reboot is required use the needs-restarting program. needs-restarting is available in the dnf-utils or yum-utils package.

Your bash script would look something like this.

if [[ $(needs-restarting -r &>/dev/null) -eq 1 ]]
then
    reboot_required=1
fi

if [[ ${reboot_required} -eq 1 ]];
then
    reboot
fi

If you are running your scripts part of a Systemd sevice, then you should do this instead in your bash script:

if [[ "${reboot_required}" -eq 1 ]]
then
    echo "Scheduling reboot in 5 minutes"
    # shutdown -r +5
    systemd-run --on-active=5m shutdown -r now
else
    echo "No reboot required"
fi

Also, you should check for updates at 4:30 AM Eastern Standard Time (EST). That is around the time Fedora seems to release their daily updates. If the machine is updated and rebooted early in the morning, it will be ready for users in the AM.


yum and dnf do the right thing.


To remove old kernels use the following code for yum:

# yum-utils package
if ! package-cleanup --oldkernels --count=1; then
    echo "Failed to remove old kernels"
    exit 1
fi

To remove old kernels use the following code for dnf:

old_kernels=($(dnf repoquery --installonly --latest-limit=-1 -q))
if [ "${#old_kernels[@]}" -eq 0 ]; then
    echo "No old kernels found"
    exit 0
fi

if ! dnf -y remove "${old_kernels[@]}"; then
    echo "Failed to remove old kernels"
    exit 1
fi

You should remove the old kernels after the reboot.


Stepping back to 10,000 feet, you should not be doing this stuff manually. The fact that a user must do something special - like run dnf update or apt-get update - is, in my opinion, a security bug for the distro. It is a security engineering defect. For a more detailed discussion of this problem, see Linux Mint 20.2 is a bit more insistent about updating but not as annoying as Windows or Mac on the Linux Mint GitHub.

To fix the problem you should check out my auto-update GitHub. It is a Systemd service that always updates the machine without user intervention. It also reboots the machine as required.

I wrote the scripts for auto-update because my parents and grandparents are using Linux. I cut them over from Windows to Linux several years ago. My parents and grandparents simply cannot do all the crap required to keep a Linux machine up to date. It is a complete security engineering failure on the part of Linux distros.

I am not sure I agree that not automatically updating is an engineering defect. That is a pretty strong statement that has a lot of problems. Especially in the context of this topic where the OP is trying to avoid rebooting, not have it happen automatically.

That being said, if you want to auto-update, Fedora has built-in support for this through dnf-automatic. There is no reason to use an external systemd service.

1 Like

Also see Peter Gutmann’s Engineering Security. Not updating a machine automatically violates the Defend, Don’t Ask rule. Asking a user to do something is a complete failure from the security engineering point of view. We’ve know it is a complete failure for years. We have the Security UX studies to prove it is a complete failure.

I would be curious to know what problems happen with automatic updates. I’ve seen two that are pinned in my memory. The first is a Norton Antivirus update that bricked about 200 machines on my network back in 2002 or so. The second was a Ubuntu kernel update problem back in 2015 or so.

What I can tell you is, out-of-date machines present a lot of risk. I remember the Worm Wars back in the early 2000’s when Nachia and Blaster were running rampant through Windows network. They shut down some networks for weeks while machines were being re-imaged and databases were being restored. All the networks needed were the updates Microsoft provided two months earlier.

Reboots are required whenever files are in use or a new kernel is installed. It is non-negotiable. You can check when a reboot is required by using the program needs-restarting.

One obvious challenge would be how to you avoid interrupting long-running jobs? On a workstation you can have jobs that take several days to complete. Sometimes you need to not update.

The vast majority of updates aren’t security-related. Some people just prefer to defer updates.

The reality is that good security isn’t about doing one absolute thing. It is about having a sound strategy that factors risk in your personal situation. I am pretty security-focused and have my own opinions but everyone needs to freedom to make their own risk management decisions.

As the end-user, it is negotiable. You can choose not to do it. Again, not all kernel updates are about security.

Have you actually read this topic because we covered this in several posts already. Also, not everyone has needs-restarting installed. However, dnf needs-restarting should be available to everyone.

1 Like

OTOH, I absolutely hate the forced updates and attendant restarts that are a part of the microsoft world.

While it may be an issue to not do a security update & restart, it is far from pleasant to have a system do a forced restart because of an automatic update where the system decides to do a reboot.

Case in point:
I do a lot of 3D printing, and some print jobs take a long time. I recall one recently that took more than 40 hours to complete. Had there been a forced reboot during that time it would have been a waste of time, waste of resources for the printer, and at best would have been infuriating.

Your idea that a forced update with forced reboots is a good thing can at best be considered abhorrent to those of us who take care of our systems and do updates on our own schedule.

I do not personally know any linux user who wishes their system were remotely managed without user interaction and approval such as happens with windows users.

1 Like

That sounds like a bug in the job or task. If it is sensitive or have special needs, then it should save its state during shutdown and restart the job or task after the reboot. Avoiding updates and reboots due to defective software is just compounding the problem.

That assumes are updates are labelled properly. What I’ve found is, some (many?) could be and some are mislabeled.

The problem is, a bug is “security related” if it obvious or has a working proof of concept. That is, it is not-serious unless proven otherwise. I feel that is the wrong end of the stick. Many bugs should be considered serious or security related unless it is proven otherwise.

As a maintainer and contributor to several packages, I can tell you I fix a lot of problems that probably should be considered serious or security related (and even have a CVE). But I don’t bother trying to develop a POC or get a CVE because it is a waste of my time. I’ve seen nearly every other project I monitor or contribute to do the same. Other project maintainers are in the same situation. We only have a limited number of cycles to spread among responsibilities.

The efficient use of time for maintainers is, identify the problem, fix the problem, push the fix, ask folks to test, and push a new release. Lather, rinse, repeat.

Hmmm… the Risk Management Frameworks I have worked with all place a priority on patches and updates. Patches and updates have their own line item.

An updated machine is one part of a defensive strategy. I’ve never seen a risk strategy skip the patches and updates.

Skipping the patches and updates is known to be a bad idea. Microsoft performed a study in the early 2000’s and found a machine was under attack within 30 seconds of being connected to the internet. Microsoft also found compromised machines were compromised using vulnerabilities that were 30 days or older. Patches and updates were available, they just weren’t applied.

Choice is good. If someone is savvy like you, then that is fine. You should have to do something special to get into a less secure/insecure state.

The vast majority of users are not like you. They need this stuff to just work out of the box. For the vast majority of users, they need to be in the more secure state out of the box.

A fellow I worked with at Cigital once told me, “developer driven security is some of the worse security you will encounter.” He was right. Apt, dnf, pkg, yum and friends are the embodiment of it. The package managers are insecure out of the box based on some crap developers have come up with. The developers are completely out of touch with reality. I’ll even go so far to say most of them have not read Peter Gutmann’s Engineering Security or Ross Anderson’s Security Engineering. Most developers have not reached the point they can identify the problem.

A great example of an [incorrect] assumption is, every user is savvy like you. We know they are not. Most users are not qualified to make security related decisions. Another [incorrect] assumption is, all users have buggy long running jobs or tasks that don’t save state and can’t be restarted. I seriously doubt most users have long running tasks with buggy software.

The need for a reboot is not negotiable. If a reboot is required, then it is necessary. One who chooses to forgo the reboot is running with unpatched, buggy software. One who chooses to forgo the reboot is simply making a bad decision. They probably have an equally bad excuse why.

Hello @p4ddyf
I liked this topic a lot and am catching back even if the problem was resolved
Can you tell me what dragora notifier is? I mean I don’t have such notifier in my system as far as I know… I just get update notification from the software centre


And I update it via terminal instead through the software centre…
But I’ve got a question while updating it through the software centre… It always says Reboot Required but I never have got that while updating through terminal


Can you help me with which command do we reboot after kernel and systemd updates ?
I guess just commanding reboot won’t work because it just reboots the system instead of popping up the installing update screen while rebooting and downloading


And can you tell me how to avoid some updates if we don’t want it in ?
like some nvidia removal things


how do update and install option work in linux? I mean in which case do the system has to reboot and install the updates just brought
image
I mean this kind of reboot?

Here I have syetemd and kernel updates?
According to what I have heard it requires reboot but
image

No that update and install option did not come up?
does that mean normal reboot is the go or there is some command for doing that?

Yes, a normal reboot should be sufficient to switch your system to using the updated kernel. You can check what kernel your system is running with the following command.

$ uname -r

The result of the above command should be different after you reboot if you have installed a newer kernel.

1 Like

@glb Can you tell me what sort of updates requeires This sort of reboot?
image

I think the current GNOME’s approach to installing updates is probably the best possible solution in both theoretical and practical terms, specifically for casual users.

Perhaps, the only way to improve is installing updates in a separate layer, like a Btrfs snapshot, and then just notify the users that they can reboot when ready to apply changes.

I’m afraid that any more invasive or forceful approach will just anger the users and lead to a decrease in the popularity of the distribution.

4 Likes