although the question is 2y old, the topic surely is not.
according to RFC3484 Default Address Selection for IPv6
the OS automatically selects the best IP address, which is either IPv6 or IPv4.
if /etc/gai.conf is unchanged and your DNS works fine, than Linux (and any other OS) will prefer IPv6 address(es), otherwise it will choose IPv4, to connect to e.g. time.google.com.
you can check that by
getent ahosts time.google.com
every software should try all ip addresses of that list in that order, beginning at the top. where did your linux get the timeservers from? if it is DHCP than your OS perhaps cannot choose because it got IPv4 IP adresses, not FQDNs?! the ntp client may show reverse DNS hosts to IP addresses, not the real configured ntp server "names".
/edit: I tried with ntpd 4.2.8p15 and ntpd 4.2.6p5 on opensuse15.2 and centos7. both ntpd do not obey rfc6742(3484). chronyd does btw.
HTH