When systemd services fail you usually don't really notice it unless it's a critical service. However there may be situations where it would be nice that you are mare aware of failing services.
Motivation
On my machine I have setup backup jobs using papanito.borg. This role sets up a systemd service and a systemd timers which executes the backup regularly and unless I explicitly check with systemctl status automatic-backup-clawfinger-local I will not know if the backup was successful or not. This may be fatal if I need a recent backup but the most successful backup happened month ago. So I would like to get a notification in case of a failure.
Possibilities
As the backup solution above executes a script, I might implement the notification in the script. But my focus here is a more generic approach which is suitable for other services as well. Systemd actually offers different possibilities for things to happen in case of a failure:
ExecStopPost=: Additional commands that are executed after the service is stopped - to invoke commands when a service failed to start up correctly and is shut down again.FailureAction=: Configure the action to take when the service enters a failed state.OnFailure=: A space-separated list of one or more units that are activated when this unit enters the “failed” state.
FailureAction and ExecStopPost
FailureAction seems the better choice for my initial goal - notify in case of failures. In conjunction with notify-send one could send notifications directly to the desktop:
[Service]
Type=simple
ExecStart=/bin/sh /opt/borg_backups/automatic-backup-clawfinger-local.sh
User=aedu
Group=aedu
FailureAction=/usr/bin/notify-send --app-name="%N" "Failure"
Unfortunately that does not work, and as I suspected, it is related to the missing graphical user session
notify-send communicates over the dbus session bus but cron jobs are not part of any graphical user session;
So following the post above, I have adjusted FailureAction as follows
FailureAction="DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus /usr/bin/notify-send --app-name="%N" 'Failure'"
However, systemd does not like it - or well I don’t know how to properly write the command in the .service-file
Jul 15 19:56:50 archlinux systemd[1]: /etc/systemd/system/automatic-backup-clawfinger-local.service:10: Neither a valid executable name nor an absolute path: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
So I’ve created a script in /usr/local/bin/systemd-notify.sh with the following content, assuming the logged in user is 1000:
#!/usr/bin/env bash
DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus notify-send --app-name=$1 $1@$(hostname) "Failed"
And the .service file updated accordingly:
FailureAction=/usr/local/bin/systemd-notify.sh "%N"
As this still does not work, I use ExecStopPost instead, which seems to work fine.
Ultimately I enhanced the script a bit so that one can also specify the user id as well as add more detailed information to the notification. Have a look at https://github.com/papanito/shell-scripts/blob/master/scripts/scripts/systemd-notify.sh
OnFailure
A nice alternative is OnFailure, which requires an additional systemd job of type oneshot, let’s call it systemd-desktop-notifier.service
[Unit]
Description=Desktop Notification of %i
[Service]
Type=oneshot
ExecStart=/usr/local/bin/systemd-notify.sh %i
User=nobody
Group=systemd-journal
You can then specify OnFailure in the .service-file of the service you want to get notifications as follows in the [Unit] section
OnFailure=systemd-desktop-notifier@%N.service
Using the %N specifier here and the %i specifier in the systemd-desktop-notifier.service, are used to properly set the notification message. The interesting thing using OnFailure is, that you can provide a command separated list which allows you to send notifications in multiple ways.
For instance, I also have a googlechat notifications - useful if you are not sitting on your computer and you get a notification. Thus I would have a seconde service systemd-googlechat-notifier.service
[Unit]
Description=Desktop Notification of %i
[Service]
Type=oneshot
ExecStart=/usr/local/bin/systemd-googlechat-notify.sh %i
User=nobody
Group=systemd-journal
Which calls systemd-googlechat-notify.sh that contains the following code
#!/usr/bin/env bash
curl --data "{'text':'*$1@$(hostname)*\nFailed'}" --header 'Content-Type: application/json; charset=UTF-8' --request POST https://chat.googleapis.com/v1/spaces/xxxxxxxxxxxxxxxxxx
You may check systemd-googlechat-notify.sh which shows an enhanced script which you can use, if you are using googlechat.