Deploying systemd Podman services to CoreOS

Hi,

I’m contemplating a small project to deploy FCOS on a rpi4 to host some container services. I have a pretty good understanding of how to install FCOS on the CF disk and include the initial set of service configurations (systemd Podman wrappers) via Ignition. I also understand I can use

  • FCOS Auto Updates to pull in OS changes
  • podman pull for container image updates

What I’m unclear about is how to deploy new services or modifications to the initial set of systemd unit files. The obvious choices I see are to

  1. re-run coreos-installer with the new files
  2. use something like Ansible
  3. install some other container manager (e.g. k3s) to manage the services
  4. other?

Re-running the installer seems like overkill since that requires me to physically rewrite the flash disk or to boot the installer over the network. Using Ansible feels like it defeats the purpose of transactional updates to the OS and containers themselves. And I like the simplicity of Podman and feel like Kubernetes, even simplified in k3s, would add unnecessary complexity.

Am I missing another obvious option? Does anyone have a recommendation for me to pursue? Thx in advance!

Hi,
If you have network boot option you can use it as a diskless install : live PXE
I do it for one of my servers

Of course I have disks, 2 disks in a raid array

I boot from pxe, download ign file, and run newly added units in the ignition file after a reboot

I don’t know if it’s a reliable solution for you, let me know I can help tou

for images/container updates you can use podman auto-update feature :

create a service like that : container-update.service

[Unit]
Description=Podman auto update containers

[Service]
Type=oneshot

ExecStart=/usr/bin/podman auto-update
ExecStartPost=/usr/bin/podman container prune --force
ExecStartPost=/usr/bin/podman image prune --all --force

The timer associated to it : container-update.timer

[Unit]
Description=daily update containers

[Timer]
OnCalendar=*-*-* 00:00:00
Persistent=true
RandomizedDelaySec=2h

[Install]
WantedBy=timers.target

and add a label --label="io.containers.autoupdate=image" and the ENV Environment=PODMAN_SYSTEMD_UNIT=%n when you run your service, here an example with grafana : grafana.service

[Unit]
Description=Podman grafana.service
Wants=network.target
After=network-online.target
PartOf=pod-monitor.service

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman run --label="io.containers.autoupdate=image" --detach --replace --name=%N --pod=pod-monitor -v /run/mount/configs/grafana:/var/lib/grafana docker.io/grafana/grafana:latest
ExecStop=/usr/bin/podman stop %N
Type=forking

Once done, every night between 00:00:00 and 02:00:00 your grafana and all container labelled with --label="io.containers.autoupdate=image" and Environment=PODMAN_SYSTEMD_UNIT=%n will be updated

Hi Emeric,

Thanks so much for those suggestions.

I had considered PXE Boot as an option. One benefit of PXE boot is that I wouldn’t need to rewrite the flash just to modify the systemd files but one obvious drawback is that I’d still need another machine to host the images themselves.

I hadn’t considered Podman auto-update but I will take a look. That does seem like a good option for managing container image versions but I don’t think that would allow me to, for example, deploy a new systemd unit to the machine or to modify an existing one. Or maybe I can do that from within a specially crafted container that I build myself?

Anyway thanks for your inputs!

brian

podman auto-update won’t help you at all creating new services, it just perfom updates on existing ones

depending the number of services you need you can just create one like /etc/systemd/system/myunit.service (default path)
enable them with systemclt enable --now myunit.service and that’s all that ignition do

you can also create a script that scp your files in /etc/systemd/system/
run systemclt enable --now myunit.service via ssh …

Thanks, that’s very helpful.

You might want to look at Fedora IoT for this. It’s based on the same basic technology as Fedora CoreOS, but with the IoT use-case in mind, including a web-based provisioning system: Setting up a Device with Zezere :: Fedora Docs

Interesting, thanks for sharing that link @mattdm . I took a look at the docs and see a reference to rpi3 installations. Do you know if Fedora IoT supports the rpi4? Thx.

The word “support” is complicated. We don’t have any RPi-specific modification in Fedora Linux in general — we want all of that kind of thing to be in the upstream kernel and tools so it benefits everyone. And not all of the RPi4 works perfectly yet — I don’t remember the details (probably a separate question, and maybe even something we could keep track of as a “FAQ” post) but there are some issues with sound and accelerated graphics, and people expect “supported” to mean “everything just works”.

But I think for your use case, it should be just fine.

Hi,

I realize my question was a little unclear. What I meant was if IoT can even be installed on the rpi4, but you raise a good point about the overall hardware support for that platform.

To clarify my confusion

It may just be that the docs are slightly out-of-date but it’s also not clear how the different spins relate in terms of hardware support. Hence my question.

Thanks!

brian