When systemd services fail you usually don't really notice it unless it's a critical service. However there may be situations where it would be nice that you are mare aware of failing services.

Motivation

On my machine I have setup backup jobs using papanito.borg. This role sets up a systemd service and a systemd timers which executes the backup regularly and unless I explicitly check with systemctl status automatic-backup-clawfinger-local I will not know if the backup was successful or not. This may be fatal if I need a recent backup but the most successful backup happened month ago. So I would like to get a notification in case of a failure.

Possibilities

As the backup solution above executes a script, I might implement the notification in the script. But my focus here is a more generic approach which is suitable for other services as well. Systemd actually offers different possibilities for things to happen in case of a failure:

• ExecStopPost=: Additional commands that are executed after the service is stopped - to invoke commands when a service failed to start up correctly and is shut down again.
• FailureAction=: Configure the action to take when the service enters a failed state.
• OnFailure=: A space-separated list of one or more units that are activated when this unit enters the “failed” state.

FailureAction and ExecStopPost

FailureAction seems the better choice for my initial goal - notify in case of failures. In conjunction with notify-send one could send notifications directly to the desktop:

[Service]
Type=simple
ExecStart=/bin/sh /opt/borg_backups/automatic-backup-clawfinger-local.sh
User=aedu
Group=aedu
FailureAction=/usr/bin/notify-send --app-name="%N" "Failure"

Unfortunately that does not work, and as I suspected, it is related to the missing graphical user session

notify-send communicates over the dbus session bus but cron jobs are not part of any graphical user session;

So following the post above, I have adjusted FailureAction as follows

FailureAction="DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus /usr/bin/notify-send --app-name="%N" 'Failure'"

However, systemd does not like it - or well I don’t know how to properly write the command in the .service-file

Jul 15 19:56:50 archlinux systemd[1]: /etc/systemd/system/automatic-backup-clawfinger-local.service:10: Neither a valid executable name nor an absolute path: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus

So I’ve created a script in /usr/local/bin/systemd-notify.sh with the following content, assuming the logged in user is 1000:

#!/usr/bin/env bash
DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus notify-send --app-name=$1 $1@$(hostname) "Failed"

And the .service file updated accordingly:

FailureAction=/usr/local/bin/systemd-notify.sh "%N"

As this still does not work, I use ExecStopPost instead, which seems to work fine.

Ultimately I enhanced the script a bit so that one can also specify the user id as well as add more detailed information to the notification. Have a look at https://github.com/papanito/shell-scripts/blob/master/scripts/scripts/systemd-notify.sh

OnFailure

A nice alternative is OnFailure, which requires an additional systemd job of type oneshot, let’s call it systemd-desktop-notifier.service

[Unit]
Description=Desktop Notification of %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/systemd-notify.sh %i
User=nobody
Group=systemd-journal

You can then specify OnFailure in the .service-file of the service you want to get notifications as follows in the [Unit] section

OnFailure=systemd-desktop-notifier@%N.service

Using the %N specifier here and the %i specifier in the systemd-desktop-notifier.service, are used to properly set the notification message. The interesting thing using OnFailure is, that you can provide a command separated list which allows you to send notifications in multiple ways.

For instance, I also have a googlechat notifications - useful if you are not sitting on your computer and you get a notification. Thus I would have a seconde service systemd-googlechat-notifier.service

[Unit]
Description=Desktop Notification of %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/systemd-googlechat-notify.sh %i
User=nobody
Group=systemd-journal

Which calls systemd-googlechat-notify.sh that contains the following code

#!/usr/bin/env bash
curl --data "{'text':'*$1@$(hostname)*\nFailed'}" --header 'Content-Type: application/json; charset=UTF-8' --request POST  https://chat.googleapis.com/v1/spaces/xxxxxxxxxxxxxxxxxx

You may check systemd-googlechat-notify.sh which shows an enhanced script which you can use, if you are using googlechat.