systemd service fails to use `tail` to output to console

Question

I fail to configure my systemd service properly (systemd 249.11-0ubuntu3.12). It should start a tail process which outputs to the console. Ubuntu running as Proxmox VM.

When booting, the service is started but exits without helpful output:

× watchbackup.service - Startup Applications
     Loaded: loaded (/etc/systemd/system/watchbackup.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2024-03-01 21:47:01 CET; 27s ago
    Process: 939 ExecStart=/opt/backup-proxmox/tools/watchbackup.sh (code=exited, status=1/FAILURE)
   Main PID: 939 (code=exited, status=1/FAILURE)

If I restart the service manually, it works as intended. The execed shell script (I failed miserably to put the tail directly to the service unit file's ExecStart command):

#!/bin/sh
sleep 5
ls -la  /net/var/archiv-1/log/ > /tmp/foo
echo tty1 > /dev/tty1
echo tty0 > /dev/tty0
echo console > /dev/console
tail -f /net/var/archiv-1/log/backup.log 

The sleep succeeds to delay, the file exists as expected (although /net/var/archiv-1 is an NFS mount), all three echoes are visible on the console. However, tail terminates immediately.

My last attempt service unit file looks like this:

[Unit]
Description=Startup Applications
After=network.target
After=net-var-archiv\x2d1.mount
After=net-var-archiv\x2d1.service

[Service]
WorkingDirectory=/opt/backup-proxmox
ExecStart=/opt/backup-proxmox/tools/watchbackup.sh
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Type=simple
StandardOutput=tty
StandardError=tty
TTYPath=/dev/tty1

[Install]
WantedBy=multi-user.target

Any suggestions to identify the reason why I cannot tail the file if the service is started at boot time but I can when restarting the service manually?

Answer 1

Use the Native SystemD Journal

Replace tail with journalctl -f -u watchbackup.service

I can't find out why tail isn't allowed, only the fix. I'll add a more fleshed out answer when I can find why, but I suspect it has something to do with the fact that tail cannot monitor a real time write, and systemd being the init system is running in real-time.