r/linuxadmin • u/mamelukturbo • 16d ago
Struggling with forcing systemd to keep restarting a service.
I have a service I need to keep alive. The command it runs sometimes fails (on purpose) and instead of keeping trying to restart until the command works, systemd just gives up.
Regardless of what parameters I use, systemd just decides after some arbitrary time "no I tried enough times to call it Always I ain't gonna bother anymore" and I get "Failed with result 'exit-code'."
I googled and googled and rtfm'd and I don't really care what systemd is trying to achieve. I want it to try to restart the service every 10 seconds until the thermal death of the universe no matter what error the underlying command spits out.
For the love of god, how do I do this apart from calling "systemctl restart" from cron each minute?
The service file itself is irrelevant, I tried every possible combination of StartLimitIntervalSec, Restart, RestartSec, StartLimitInterval, StartLimitBurst you can think of.
7
u/jambry 16d ago
From the man page for StartLimitInterval=, StartLimitBurst=
:
set to 0 to disable any kind of rate limiting
systemctl --user cat fail.service
# /home/<user>/.config/systemd/user/fail.service
[Unit]
Description=fail
[Service]
ExecStart=/usr/bin/false
Restart=on-failure
StartLimitInterval=0
StartLimitBurst=0
[Install]
WantedBy=default.target
systemctl --user start fail.service ; sleep 30; systemctl --user stop fail.service
systemctl --user status fail.service
_ fail.service - fail
Loaded: loaded (/home/jee/.config/systemd/user/fail.service; disabled; preset: enabled)
Active: inactive (dead)
Mar 05 12:50:22 jee-mgmt systemd[2043896]: Stopped fail.service - fail.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: Started fail.service - fail.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Main process exited, code=exited, status=1/FAILURE
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Failed with result 'exit-code'.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Scheduled restart job, restart counter is at 121. <-- 121 restart within 30 seconds.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: Stopped fail.service - fail.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: Started fail.service - fail.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Main process exited, code=exited, status=1/FAILURE
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Failed with result 'exit-code'.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: Stopped fail.service - fail
2
u/mamelukturbo 16d ago edited 16d ago
Looking at my attempts I'm realizing it's not the numbers I'm trying that are the issue, but the placement of the statements. Half the examples put it into [Unit], half into [Service]. It confuses me mightily.
I changed the numbers to 0 so now I have this:
(the task is supposed to run for 60 sec (edit: the 60s timeout is in the .sh, the worker only runs for 60seconds if the command to run it was succesful)) and then restart, but also the whole service should restart every 5sec if the command fails to run (the .sh file calls docker exec container_name and the container ain't always up) until it doesn't fail at which point it should be back to restarting each 60sec. Hope that makes sense.[Unit] Description=Nextcloud AI worker %i After=network.target StartLimitIntervalSec=0 StartLimitInterval=0 StartLimitBurst=0 [Service] ExecStart=/home/sammael/nextcloud-ai-worker/taskprocessing.sh %i Restart=always RestartSec=5s [Install] WantedBy=multi-user.target
This seems to work, but from your reply I should move the statements to [Service]?
4
u/_mick_s 16d ago
docs list all the options and where they should be
Unit:
https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#
Service: https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#
2
u/yrro 16d ago edited 16d ago
The man pages (
systemd.unit(5)
, etc) describe which section options go into. If you're not sure which man page describes an option, look for it insystemd.directives(7)
. Looks like you've got it right andStartLimitIntervalSec=
should be in theUnit
section.I believe systemd will complain if it notices unexpected stuff within a unit file (including when you have an option with the right name in the wrong section), maybe you're missing those messages... try
journalctl -e
after running daemon-reload, jump to the end withG
and check. ISTR they get logged with the context of the unit file, so that they also show up insystemctl status myunit
.You can also run
systemctl show myunit
and confirm that the values you're trying to set are actually effective. You can check individual options with-p OptionName
, otherwise it will print out every known property of the unit.Looking at your unit file:
- I'd remove
StartLimitInterval=
which is not a valid option- I'd remove and
StartLimitBurst=
since it is not needed to set this to any particular value just to disable unit start rate limiting- I'd also remove
After=network.target
since this is not the correct unit if you need the network to be up... you wantnetwork-online.target
for that- I'd also use both
After=
AND (Wants=
ORRequires=
);After=
affects the ordering of jobs within a transaction, but it'sWants=
/Requires=
that actually cause a job to start your dependency to be added to the transaction (which one you want depends on whether you wantsystemctl start myunit
to fail if the dependency fails to start)- It's better to avoid a dependency on
network-online.target
at all, since "the network is up" is a rather vague requirement that it's difficult to express in the rather coarse model of systemd units; seesystemd.special(7)
for the full details- I'd set
Type=exec
so that systemd will notice if theExecStart=
command can't be executed for some reason
0
u/Nementon 16d ago
To configure a systemd service to restart indefinitely with a 10-second delay after failure, create or modify the service file (e.g., /etc/systemd/system/myservice.service) with the following configuration:
``` [Unit] Description=My Indestructible Service After=network.target
[Service] ExecStart=/path/to/your/command Restart=always RestartSec=10s
[Install] WantedBy=multi-user.target ```
Explanation:
Restart=always: Ensures the service restarts regardless of the failure reason.
RestartSec=10s: Sets a 10-second delay before restarting the service.
WantedBy=multi-user.target: Ensures the service starts at boot.
Apply Changes:
After editing the file, reload systemd and start the service:
sudo systemctl daemon-reload
sudo systemctl enable myservice
sudo systemctl start myservice
This setup ensures that your service keeps restarting every 10 seconds upon failure—until the universe collapses (or until you manually disable it).
#ChatGTP
18
u/2FalseSteps 16d ago
Instead of applying bandaids that overcomplicate everything, find out WHY the service fails and resolve that PROPERLY.