Time drift and wrong timezone within NTP configuration

Question

We have 2 VLANs, each VLAN has a server that provides DHCP, DNS and NTP to its corresponding VLAN. These 3 servers take time from a local NTP server. Here is the configuration of NTP client and servers on each VLAN and the problem of each setup:

VLAN 1:

NTP Server (Scientific Linux 7.3)

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
logfile /var/log/ntp.log
driftfile /var/lib/ntp/drift
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
restrict 127.0.0.1
restrict xx.xx.xx.xx mask XX.XX.XX.XX nomodify notrap

NTP Client (Scientific Linux 7.3)

driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
server yy.yy.yy.yy   # added by /sbin/dhclient-script

This server is always drifting 3 hours forward despite the correct timezone.

VLAN 2:

NTP Server (Scientific Linux 6.4)

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
logfile /var/log/ntp.log
server 127.1.1.0 # local clock
fudge  127.127.1.0
driftfile /var/lib/ntp/drift
restrict 127.0.0.1
restrict xx.xx.xx.xx mask XX.XX.XX.XX nomodify notrap

NTP Client (Scientific Linux 6.4)

driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
server yy.yy.yy.yy   # added by /sbin/dhclient-script

Here I could not change the time zone. It should be EET but it is always EEST despite the /etc/localtime link value.

I have made sure that:

1. No firewall rules (iptables -F)
2. ntpd daemon is running and enabled (chckonfig on)
3. CentOS 7.x uses another time service called chronyd which blocks ntpd from startup. So I disabled it (Lost the source for this :( ).

After setting up these services, we synchronized them with our local NTP once. Through the DHCP configuration on each server we have option ntp-servers xx.xx.xx.xx; so the NTP information is distributed withing DHCP. I tried adding the local NTP server's address to the server's /etc/ntp.conf but the problem is still there.

Note that all servers are virtualized through VMWare ESXi.

Answer 1

If your /etc/localtime is pointing to /usr/share/zoneinfo/EET, that timezone definition includes European Summer Time (= European DST) and between the last Sunday of March and the last Sunday of October, that timezone will be listed as EEST, not EET. And as of this writing, the changeover day was just yesterday...

You should read VMware timekeeping best practices for Linux VMs. In short:

• make sure the VMware Host is using correct time and timezone.
• if you're using NTP on guests:
  • make sure VMware Tools time synchronization is disabled
  • add tinker panic 0 as the first line of /etc/ntp.conf
  • if there is a local clock definition like server 127.127.1.0 in ntp.conf, comment it out.
  • put the hostnames or IP addresses of your NTP servers also into the /etc/ntp/step-tickers file to make your systems jump their clocks into correct time at boot as soon as network interfaces are activated and anything time-sensitive like databases has not yet been started.

Some pitfalls I've encountered:

• /etc/adjtime specifies what's the hardware clock expected to use: either UTC or LOCAL. Changing just the variable in /etc/sysconfig/clock won't necessarily do what you want. Make sure both places are in agreement, just to be safe.
• make sure your NTP servers are serving correct UTC time.
• when a system time is an exact number of hours off, first use date -u to verify that the system's idea of UTC time is correct. NTP only ever deals with UTC; any conversion error to local time is the fault of timezone settings.

Answer 2

So after a struggle with these servers the issue has been resolved. Here are the details:

1. For the CentOS 7.X server, I disabled ntpd and used chronyd instead while maintaining ntpd on the clients. It seems that chronyd will be the supported NTP daemon from CentOS 7.X onwards. chronyd configuration requires only server xx.xx.xx.xx prefer iburst and allow yy.yy.yy.yy/ZZ in /etc/chrony.conf
2. For the CentOS 6.4 server I had to install new version of tzdata here and then link /etc/localtime to the corresponding timezone, in my case Asia/Amman

Some pitfalls I encounted:

• The nightmare error no server suitable for synchronization found on the CentOS 6.4 clients was resolved by stopping ntpd, running ntpdate, then starting ntpd.
• For some reason I don't remember, in the CentOS 6.4 I had to change restrict 127.0.0.1 to restrict localhost