Program starts from rc.local but doesn't do anything

Question

I have a program which uses msqllib. It scans some hardware that I build and then updates a mysql database. This program has been running for a number of years but I have never been able to autostart it when the system reboots. Now I have an issue with power cycling that is beyond my control, so want to get it running automaticaly.

In rc.local : I have:

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Print the IP address
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
  printf "My IP address is %s\n" "$_IP"
fi
/home/nigel/scan
exit 0

Now, the code runs fine when I enter it at a command prompt, and updates my database. However, when I put it in rc.local, ps -ef shows:

root       356   350  0 20:54 ?        00:00:03 /home/nigel/scan

but the program is not executing the SQL calls to update the database.

If I ask for the statius of rc.local:

root@Pi-Scan:~# systemctl status rc-local.service
● rc-local.service - /etc/rc.local Compatibility
   Loaded: loaded (/lib/systemd/system/rc-local.service; enabled-runtime; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/rc-local.service.d
           └─debian.conf
           /etc/systemd/system/rc-local.service.d
           └─ttyoutput.conf
   Active: activating (start) since Sat 2023-05-06 20:54:19 EDT; 12min ago
     Docs: man:systemd-rc-local-generator(8)
Cntrl PID: 350 (rc.local)
    Tasks: 2 (limit: 414)
      CPU: 3.903s
   CGroup: /system.slice/rc-local.service
           ├─350 /bin/sh -e /etc/rc.local start
           └─356 /home/nigel/scan

May 06 20:54:21 Pi-Scan rc.local[356]: error: first parameter is not a valid address family: Transport endpoint is not connected
May 06 20:54:22 Pi-Scan rc.local[356]: error: first parameter is not a valid address family: Transport endpoint is not connected
May 06 20:54:23 Pi-Scan rc.local[356]: error: first parameter is not a valid address family: Transport endpoint is not connected
May 06 20:54:24 Pi-Scan rc.local[356]: error: first parameter is not a valid address family: Transport endpoint is not connected
May 06 20:54:25 Pi-Scan rc.local[356]: error: first parameter is not a valid address family: Transport endpoint is not connected
May 06 20:54:26 Pi-Scan rc.local[356]: error: first parameter is not a valid address family: Transport endpoint is not connected
May 06 20:54:27 Pi-Scan rc.local[356]: error: first parameter is not a valid address family: Transport endpoint is not connected
May 06 20:54:28 Pi-Scan rc.local[356]: error: first parameter is not a valid address family: Transport endpoint is not connected

However if I issue systemctl restart rc-local.service The program executes a second instance and works fine!

Thinking that this may be a racing condition, I have tried adding a sleep 5 before the command and still get the same problems.

Answer 1

Note that the comment

# This script is executed at the end of each multiuser runlevel.

might be no longer completely true.

It was true when classic SysVinit was in use and all the start-up scripts were run strictly sequentially. But after the startup scripts were first reworked to add more parallelism, and especially after SysVinit was replaced by systemd, it is no longer guaranteed that rc.local is executed at the end of the start-up process.

Unless systemd is explicitly told that rc-local.service should start only after your database is running, it is possible that it gets executed as soon as networking is up and running (as /usr/lib/systemd/system/rc-local.service.d/debian.conf typically has the ordering constraint After=network-online.target).

Since your MySQL database most likely has the same ordering constraint, it is likely that your rc-local.service and the database are started in parallel at the same time, effectively racing against each other. And since starting a database can easily take more than 5 seconds (especially on a RasPi), it is likely that rc-local.service will "win" the race, and will attempt to connect to the database when the database is not fully running yet.

It also seems to me that your /home/nigel/scan might benefit from improvements in its error handling code. The repeated Transport endpoint is not connected errors suggest that your program perhaps tried to open a connection to the database, ignored the fact that it failed, and then keeps retrying to send some data over a connection that is not successfully opened in the first place: an attempt which is already doomed to fail. The part that primarily needs retrying is opening the connection, not sending the data.

Answer 2

To extend on telcoM's excellent answer (which you imho should at least upvote, and accept!):

The most straightforward way to solve this is by not using the ancient rc.local mechanism, but creating a hardware-scan service file, and making it depend on the database server having already started. Then you get the guarantee that exactly what you need is started as soon, but not sooner, than its dependencies are ready.

[Unit]
Description=Scan the hardware using "scan"
# I'm assuming the service is called mariadb.service, but you can check that:
# systemctl status mariadb
# should show it running; if it says that the service couldn't be found,
# try mysql.service instead (I really don't know what you've installed)
After=mariadb.service

[Service]
Type=oneshot
ExecStart=/home/nigel/scan

[Install]
WantedBy=multi-user.target

Place as hardware-scan.service in /etc/systemd/system/, and run sudo systemctl enable hardware-scan to make it start on next boot.

Note that it'll be run as root!