0

I want to set up a high availability system, and to that end, was experimenting with several recovery methods.

Assuming I have received a kernel update through automated means, which would mean a reboot, I would create a snapshot of the running system, create a bootable entry in GRUB2, update my kernel, and force a boot with the updated root.

If boot did not go through as expected, I'd want a timer which would, after a set time, abandon the boot attempt and boot normally, which would be through the pre-update snapshot.

The scenario looked rile for a WatchDog timer (WDT). I noticed that there was a Watchdog in my BIOS ACPI settings, with some time values from 3s to 30 min. I set it to 2 min, and in my boot attempt, I interrupted the boot process so it would not go ahead, sure enough, after 2 min, it rebooted. However, even if the system did boot correctly, it still kept rebooting every 2 min.

I read, here, here and here that there needs to be a daemon which should tickle / refresh this watchdog. I noticed that there is a /dev/watchdog in my system, however, I have no idea if this is the watchdog which interfaces with the watchdog I enabled in BIOS. I further read that this can be tickled by systemd. Setting the RuntimeWatchdogSec= option in /etc/systemd/system.conf should send a refresher to the watchdog timer and prevent it from rebooting. Accordingly I set this to 20s, but still my device keeps on booting whenever the set timer frequency in BIOS watchdog elapses. (For some reason I have /dev/watchdog and /dev/watchdog0, both 0 bytes, not sure if that is normal...)

Do I need to enable something else? Is my understanding correct that the /dev/watchdog interfaces with the BIOS watchdog timer which I had enabled, and that it can be tickled by /etc/systemd/system.conf option? Initially I assumed it was a software watchdog not caring about the hardware watchdog I turned on in BIOS, but seems it should work with that.

The board I am using is a very generic board, and OS is CentOs 8.

EDIT:

Doing a lsmod | grep wdt gives me the following :

iTCO_wdt 16384 1
iTCO_vendor_support 16384 1 iTCO_wdt
mei_wdt 16384 0
mei 110592 3 mei_wdt,mei_me

Since systemd was not able to work, I downloaded the watchdog daemon provided by CentOs, and set values such as watchdog-device to /dev/watchdog, and tried setting some stuff, but that didn't work either. The system just keeps restarting.

I did a systemctl status watchdog.service, it gives me a status saying daemon service is running and is active, alongwith :

alive=/dev/watchdog heartbeat=[none] to=root no_act=no force=no
hardware watchdog identity : iTCO_wdt
cannot set scheduler (errno=1 = 'Operation not permitted') //ERROR

The error I checked might have something to do with systemd, but I checked and /etc/system.conf is completely commented.

Improve this question
2
  • If you run lsmod | grep wdt, what do you get? Commented Jun 17, 2020 at 21:30
  • @telcoM Apologies for such a late reply. I was waiting for some access to this system. I have updated my question with lsmod output, and some other information. Commented Jun 22, 2020 at 17:03

1 Answer 1

Reset to default
1

Your system seems to have at least one, but possibly two hardware watchdogs built in.

iTCO_wdt is the driver for a watchdog function built into most modern Intel chipsets. mei_wdt is the driver for another watchdog, built into Intel Active Management Technology (AMT) Management Engine Interface (MEI). It is less well-tested compared to iTCO_wdt.

It is possible that your BIOS will reserve one of the watchdogs if you enable it in BIOS settings. In that case, it would probably require some vendor-specific method to refresh it. Maybe through ACPI / WMI calls?

Seeing two /dev/watchdog* devices is normal for a single watchdog. /dev/watchdog is the classic watchdog device node: character device node with major number 10 and minor number 130. /dev/watchdog0 is the newer device node, with dynamically-determined major device number: it allows the generation of multiple watchdog devices, like /dev/watchdog0, /dev/watchdog1 etc. if the system has more than one usable watchdog device. The classic interface allows for just one watchdog device on the system.

The fact that systemctl status watchdog.service indicates it is successfully using iTCO_wdt and the module's use count in the lsmod listing is 1 seems to suggest that this watchdog would be available to use with Linux.

Your BIOS, on the other hand, may be using mei_wdt, which would explain why the watchdog.service and systemd failed to refresh the watchdog and avoid the reboot. Also, the fact that /dev/watchdog1 did not appear suggests that the mei_wdt driver failed to initialized the watchdog function of the MEI - perhaps because the BIOS is already using it, and does not allow it to be used directly by the operating system?

You might try disabling the watchdog setting in BIOS and just using the iTCO_wdt watchdog in Linux: the driver for that watchdog should be perfectly capable for activating that watchdog on its own, unless the BIOS plays tricks with hardware configuration. In this way, you might miss protection against boot-time faults, but the watchdog should still protect you if the system hangs up when Linux is running.

When nothing else is using the watchdog device, you should be able to use the wdctl command to get information about the current status of the watchdog device.

Improve this answer
2
  • Thank you for your answer. I understand perhaps a bit better now. So, assuming BIOS has reserved mei_WDT for its enabled watchdog, and since the driver isn't loaded (RHEL/centos has the driver according to documentation), because BIOS is exclusively(??) using it, I can't understand its purpose, what is it supposed to be watching and how do you inform it that everything is OK? This is a very generic off-the-shelf PC, can't imagine the need for a specialized driver for a watchdog nobody can touch otherwise. For my purpose I need a WDT which can react during boot hang, not from a running system. Commented Jun 22, 2020 at 19:51
  • May I further ask, how do I verify all this? I suppose I can ask the BIOS vendor what it actually uses, but from my own side, what can I do with the OS to actually confirm that this is what is indeed happening? Also, is there a way to somehow force the mei_WDT to show up or force load it and configure ? Commented Jun 22, 2020 at 19:56

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.