0

Our build team just spun up two RHEL 9.6 VM. The ECDSA fingerprint always resolves to the same value when I ssh into other servers. I've never seen this before. What could be causing it?

$ ssh -V
OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024
$ ssh -q bboptappgs402b
The authenticity of host 'bboptappgs402b (10.143.170.85)' can't be established.
ECDSA key fingerprint is SHA256:<redacted>
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:1: bbopcdspgs402a
    ~/.ssh/known_hosts:3: bbopcdspgs402b
    ~/.ssh/known_hosts:4: bboptappgs402a
Are you sure you want to continue connecting (yes/no/[fingerprint])? no

When ssh'ing one of the new servers from an existing server, I get a similar error.

$ ssh BBOPITCPGS402A
The authenticity of host 'bbopitcpgs402a (10.143.170.87)' can't be established.
ECDSA key fingerprint is SHA256:HuTbFO0Xk+ciBO9af6FGv1DhUw6UpH1OjtXip3Y8gUk.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:1: bbopcdspgs402a
    ~/.ssh/known_hosts:2: bbopcdspgs402b
    ~/.ssh/known_hosts:3: bboptappgs402a
    ~/.ssh/known_hosts:4: bboptappgs402b
Are you sure you want to continue connecting (yes/no/[fingerprint])? no
Host key verification failed.
Improve this question
7
  • How were these machines provisioned? And are the bare metal installs or something else? Commented Sep 12 at 18:52
  • @LvB they're VMware VMs. I don't know the specifics of what the Build Engineer did. Commented Sep 12 at 18:59
  • 1
    I suspect there just restored images and not fresh installs. And the “Build engineer” has “forgotten” to reroll the machine identifiers and keys (a required step when doing such installs). Possibly cloudinit was intended to be used but it wasn’t configured properly and therefore it never fired. (I have seen that a few times). Commented Sep 12 at 19:06
  • @LvB ask him "please reroll the machine identifiers and keys"? Commented Sep 12 at 19:48
  • 1
    The fingerprint uniquely identifies a key. Two fingerprints are the same if and only if the keys are the same. Whether this was done on purpose or by accident – we don't know. Ask the person(s) who set up the VMs. In any case, this sounds more like a sysadmin or workplace problem, not a security issue. Commented Sep 12 at 21:03

2 Answers 2

Reset to default
2

As noted by Steffen Ullrich, you are using the same server key on all the servers, which is bad.¹

The "This host key is known by the following other names/addresses:" check was added in 2020 (d5d05cdb), released in openssh 8.5 in March 2021).

RHEL 8 came with an earlier openssh version, RHEL 9 does contain the code that checks if the host key is known by another name. So if you hadn't connected to a new machine since updating the client to 8.5 or later, you wouldn't have seen it.

The server host keys are stored in /etc/ssh/ssh_host_*,² the ecdsa host key will be on /etc/ssh/ssh_host_ecdsa_key (only readable by root) and a copy of the public key at /etc/ssh/ssh_host_ecdsa_key.pub readable by everyone.

You can get the fingerprint for any of those files with:

ssh-keygen -l -f <filename>

Since the md5 of those files is the same across all the servers, that means all the servers share the same host keys.

We can also see in the question that the servers bbopitcpgs402a, bbopcdspgs402a, bbopcdspgs402b, bboptappgs402a, bboptappgs402b (presumably different) all have the same key.

If an initial machine was installed, generating the ssh host keys, and then saved as a master image, the host key file would be the same for all servers. Instead, the ssh host keys should not be present in the VM template, but created on the individual VM (so each gets its own key). This is sometimes done by cloud-init, sometimes by the ssh starting scripts, sometimes by a systemd sshdgenkey.service. It may be hard to convince a machine that it is a first boot (which may be needed to regenerate their specific keys).

PS: you probably have the contents of /etc/machine-id repeated across machines as well

¹ The goal of the host key is to ensure that you are connecting to the right machine. If all the servers share the same host key, any of the machines could impersonate any other without being noticed. For example, a compromised "low-value" machine could "rob" the ip³ of an high-value one, and receive the ssh connections with the other side being able to detect it (since the compromised host knows the private key of the victim server).

² Unless the team building the machine explicitly configured it in a non-standard way, but given that they failed to ensure each machine has its own host keys, it seems clear they won't have changed it (HostKey directive in /etc/ssh/sshd_config).

³ Your network architecture might not allow that, but it is nonetheless a bad idea.

Improve this answer
1
  • The /etc/machine-id values of the six server are all different, but the /etc/ssh/ssh_host_ecdsa_key and /etc/ssh/ssh_host_ecdsa_key.pub are identical. These are the first RHEL9 servers that I've had to ssh from one to another, so just noticed it. Interestingly, neither putty nor SecureCRT complained about the identical ECDSA keys. (Well, they might have but with GUIs you always click Accept.) Commented Sep 14 at 10:22
0

... just spun up two RHEL 9.6 VM

Not much context is provided with this statement. But I would assume that these VM are based on the same image and that the host key is part of the image. The host key is only specific to a machine if its generated on the machine and never used on any other, i.e. it is not derived from or somehow interconnected with unique properties of the (virtualized) hardware.

Improve this answer
6
  • "I would assume that these VM are based on the same image". Almost certainly. Commented Sep 12 at 19:00
  • "that the host key is part of the image." Where do I find that? (I tried some Googling, but not sure what to look for. ssh-keyscan hostname -f` -H` for example, shows the same keys, even on systems where I'm not having this problem. Commented Sep 12 at 19:06
  • Maybe don’t scan your local host to find a diffrent result… Commented Sep 12 at 19:07
  • @RonJohn: host keys are commonly stored in /etc/ssh/ssh_host_* on the SSH server system Commented Sep 12 at 19:32
  • 2
    @RonJohn: Unfortunately I don't know what you are really looking at and what exact results you see. You might be looking at the wrong files, there might be different key files involved for different sites ... Still, the explanation should be true anyway: you'll see the same fingerprint on multiple servers if the same host key is used on these servers. As for how this happened one can only speculate. Commented Sep 12 at 20:06

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.