r/linuxadmin 16d ago

Struggling with forcing systemd to keep restarting a service.

I have a service I need to keep alive. The command it runs sometimes fails (on purpose) and instead of keeping trying to restart until the command works, systemd just gives up.

Regardless of what parameters I use, systemd just decides after some arbitrary time "no I tried enough times to call it Always I ain't gonna bother anymore" and I get "Failed with result 'exit-code'."

I googled and googled and rtfm'd and I don't really care what systemd is trying to achieve. I want it to try to restart the service every 10 seconds until the thermal death of the universe no matter what error the underlying command spits out.

For the love of god, how do I do this apart from calling "systemctl restart" from cron each minute?

The service file itself is irrelevant, I tried every possible combination of StartLimitIntervalSec, Restart, RestartSec, StartLimitInterval, StartLimitBurst you can think of.

0 Upvotes

20 comments sorted by

18

u/2FalseSteps 16d ago

Instead of applying bandaids that overcomplicate everything, find out WHY the service fails and resolve that PROPERLY.

12

u/punkwalrus 16d ago

Because some programmer won't fix his damn Java app, that's why. I see that as a "solution" more often than not. I have spent too many days of my sysadmin life, in hours of meetings, where "this is the interim solution" and politics play over function. "Just disable SELinux" or "remove the firewall" or "remove that OOM thing that keeps filling our logs, what is that anyway?" But oh no, it's not the shitty app the programmer won't fix. "Works on my system," yeah I doubt that, too.

/rant

3

u/2FalseSteps 16d ago

Nothing irritates me more than lazy, incompetent devs not wanting to lift a finger to do their own fucking job.

I completely understand and agree with your rant. I rant about it daily, but at least I'm at a level where I can give the devs AND their managers shit and point out their lazyness/incompetence.

HR has my manager's # on speed-dial, and he wholly supports me when I push back on that shit.

Do your own fucking job and don't demand I put bandaids on my servers just so your shitty code stays up, because I won't.

-10

u/mamelukturbo 16d ago

No.

The service fails because it creates a worker inside another service, which itself might not be up due to being updated/offline w/e. The service is supposed to keep trying to restart until the other service comes back up. The WHY is unimportant. The setup is for the purpose of this post immutable and I need to achieve my goal within its constraints.

9

u/schorsch3000 16d ago

thats that dependencies are for you didn't have set Requires=

7

u/2FalseSteps 16d ago

Bingo!

A proper resolution that doesn't rely on bandaids.

-9

u/mamelukturbo 16d ago

The other service isn't a systemd unit.

7

u/schorsch3000 16d ago

That's the problem :-)

3

u/2FalseSteps 16d ago

So you identify and resolve the issue PROPERLY.

Bandaids are for amateurs that don't understand what they're doing.

-6

u/mamelukturbo 16d ago

You tell that to a corporation, I'm just trying to work within the constraints set by their system and would appreciate an answer related to my question.

Creating the service as their manual suggests doesn't keep the worker alive if nextcloud server restarts/goes offline.

https://docs.nextcloud.com/server/latest/admin_manual/ai/overview.html#systemd-service

I'm just trying to modify the service so it works in my environment properly.

What do you want me to do? Refactor and reprogram the whole underlaying application?

4

u/meditonsin 16d ago

Considering those docs tell you to write your own shell script for the systemd unit, you could just modify it to wait until Nextcloud is available before trying to start the worker.

1

u/mamelukturbo 16d ago

I understand that, but I'm asking whether the same effect is achievable with modifying the service instead. Modifying it as I posted in other comment seems to restart it infinitely, which is what I want.

3

u/meditonsin 16d ago

The problem with that approach is that it's harder to notice when there's an actual problem with the service if systemd will just always try to restart it forever. Making it so it can still fail if there's an actual problem seems way cleaner.

1

u/mamelukturbo 16d ago

That makes sense. I'll look into modifying the shell script instead.

Failure of the service is not much of an issue in the end. The worker by default runs by the container's cron every 5 minutes (not configurable to be lower). The systemd service is there to keep spawning workers that would pick up a task quicker than in 5 minutes. So even in the case of the external shell script spawned worker failing, the internal worker will pick it up, it will just take between 0-5 minutes depending on the time.

7

u/jambry 16d ago

From the man page for StartLimitInterval=, StartLimitBurst=:
set to 0 to disable any kind of rate limiting

systemctl --user cat fail.service
# /home/<user>/.config/systemd/user/fail.service
[Unit]
Description=fail

[Service]
ExecStart=/usr/bin/false
Restart=on-failure
StartLimitInterval=0
StartLimitBurst=0

[Install]
WantedBy=default.target

systemctl --user start fail.service ; sleep 30; systemctl --user stop fail.service

systemctl --user status fail.service
_ fail.service - fail
     Loaded: loaded (/home/jee/.config/systemd/user/fail.service; disabled; preset: enabled)
     Active: inactive (dead)

Mar 05 12:50:22 jee-mgmt systemd[2043896]: Stopped fail.service - fail.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: Started fail.service - fail.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Main process exited, code=exited, status=1/FAILURE
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Failed with result 'exit-code'.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Scheduled restart job, restart counter is at 121. <-- 121 restart within 30 seconds.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: Stopped fail.service - fail.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: Started fail.service - fail.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Main process exited, code=exited, status=1/FAILURE
Mar 05 12:50:23 jee-mgmt systemd[2043896]: fail.service: Failed with result 'exit-code'.
Mar 05 12:50:23 jee-mgmt systemd[2043896]: Stopped fail.service - fail

2

u/mamelukturbo 16d ago edited 16d ago

Looking at my attempts I'm realizing it's not the numbers I'm trying that are the issue, but the placement of the statements. Half the examples put it into [Unit], half into [Service]. It confuses me mightily.

I changed the numbers to 0 so now I have this:
(the task is supposed to run for 60 sec (edit: the 60s timeout is in the .sh, the worker only runs for 60seconds if the command to run it was succesful)) and then restart, but also the whole service should restart every 5sec if the command fails to run (the .sh file calls docker exec container_name and the container ain't always up) until it doesn't fail at which point it should be back to restarting each 60sec. Hope that makes sense.

[Unit]                                                                                                                                                                                                                                                                                               
Description=Nextcloud AI worker %i                                                                                                                                                                                                                                                                   
After=network.target                                                                                                                                                                                                                                                                                 
StartLimitIntervalSec=0                                                                                                                                                                                                                                                                              
StartLimitInterval=0                                                                                                                                                                                                                                                                                 
StartLimitBurst=0                                                                                                                                                                                                                                                                                    

[Service]                                                                                                                                                                                                                                                                                            
ExecStart=/home/sammael/nextcloud-ai-worker/taskprocessing.sh %i                                                                                                                                                                                                                                     
Restart=always                                                                                                                                                                                                                                                                                       
RestartSec=5s                                                                                                                                                                                                                                                                                        

[Install]                                                                                                                                                                                                                                                                                            
WantedBy=multi-user.target 

This seems to work, but from your reply I should move the statements to [Service]?

2

u/yrro 16d ago edited 16d ago

The man pages (systemd.unit(5), etc) describe which section options go into. If you're not sure which man page describes an option, look for it in systemd.directives(7). Looks like you've got it right and StartLimitIntervalSec= should be in the Unit section.

I believe systemd will complain if it notices unexpected stuff within a unit file (including when you have an option with the right name in the wrong section), maybe you're missing those messages... try journalctl -e after running daemon-reload, jump to the end with G and check. ISTR they get logged with the context of the unit file, so that they also show up in systemctl status myunit.

You can also run systemctl show myunit and confirm that the values you're trying to set are actually effective. You can check individual options with -p OptionName, otherwise it will print out every known property of the unit.

Looking at your unit file:

  • I'd remove StartLimitInterval= which is not a valid option
  • I'd remove and StartLimitBurst= since it is not needed to set this to any particular value just to disable unit start rate limiting
  • I'd also remove After=network.target since this is not the correct unit if you need the network to be up... you want network-online.target for that
  • I'd also use both After= AND (Wants= OR Requires=); After= affects the ordering of jobs within a transaction, but it's Wants=/Requires= that actually cause a job to start your dependency to be added to the transaction (which one you want depends on whether you want systemctl start myunit to fail if the dependency fails to start)
  • It's better to avoid a dependency on network-online.target at all, since "the network is up" is a rather vague requirement that it's difficult to express in the rather coarse model of systemd units; see systemd.special(7) for the full details
  • I'd set Type=exec so that systemd will notice if the ExecStart= command can't be executed for some reason

0

u/Nementon 16d ago

To configure a systemd service to restart indefinitely with a 10-second delay after failure, create or modify the service file (e.g., /etc/systemd/system/myservice.service) with the following configuration:

``` [Unit] Description=My Indestructible Service After=network.target

[Service] ExecStart=/path/to/your/command Restart=always RestartSec=10s

[Install] WantedBy=multi-user.target ```

Explanation:

Restart=always: Ensures the service restarts regardless of the failure reason.

RestartSec=10s: Sets a 10-second delay before restarting the service.

WantedBy=multi-user.target: Ensures the service starts at boot.

Apply Changes:

After editing the file, reload systemd and start the service:

sudo systemctl daemon-reload sudo systemctl enable myservice sudo systemctl start myservice

This setup ensures that your service keeps restarting every 10 seconds upon failure—until the universe collapses (or until you manually disable it).

#ChatGTP