ssh won't connect with RSA key pairs iff /home directory is ceph-mounted

I've got the problem that on a server I maintain, after switching to a Ceph filesystem for the homedirectories, suddenly ssh login through RSA keys stopped working. I've tried checking several things but none of the solutions I found on the internet worked.

So to start off, yes my permissions for my homedir and .ssh/* are correct:

~$ ls -la .ssh
drwx------ 1 johndoe foo         5 10 apr 20:24 .
drwx------ 1 johndoe foo        50  8 aug 19:51 ..
-rw------- 1 johndoe foo      1617  7 aug 18:56 authorized_keys
-rw-r--r-- 1 johndoe foo       680 10 apr 20:24 config
-rw------- 1 johndoe foo      1675 17 dec  2018 id_rsa
-rw-r--r-- 1 johndoe foo       408 17 dec  2018 id_rsa.pub
-rw-r--r-- 1 johndoe foo      4050 10 apr 20:24 known_hosts

Also, it seems like every user on the server has this problem, so those permissions shouldn't be the problem.

Next thing I checked was ssh -vvv server. This shows the following output:

debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:SOMETHING /home/john/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/john/.ssh/id_dsa
debug3: no such identity: /home/john/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/john/.ssh/id_ecdsa
debug3: no such identity: /home/john/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/john/.ssh/id_ed25519
debug3: no such identity: /home/john/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

It surprised me to see that it only mentioned trying private key id_dsa as opposed to id_rsa, but after trying to connect with a server where I do not have my keys copied, it looked like it did the same thing.

I did try copying over the pubkey again with ssh-copy-id server, which asked me for my password, and then claimed I should try ssh server, but that still asks me for my password.

On the server side, I then tried checking if there were any signs of problems. So, I tried running sudo journalctl -u sshd.service -f, which gave the following output while trying to connect, then being asked for a password, and correctly entering the password:

aug 08 20:42:13 server.domain.com sshd[30191]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=sshgateway.otherdomain.com  user=johndoe
aug 08 20:42:14 octiron.liacs.nl sshd[30191]: Accepted password for johndoe from 1.3.3.7 port 43636 ssh2

So it looks like despite the key being correct, the server still gives an authentication failure.

At this point, I got stuck trying to find what I could do. The verbose output from attempting to connect with ssh clearly lists Authentications that can continue: publickey, so I'd assume pubkeys would work the way I'd expect them to. Permissions are all good, but still I can't figure out how to get the keys to work as they should.

I'd appreciate any tips to try figuring this out.

Edit: When trying to do the same with a local user (with a homedir on the local disk as opposed to a homedir on the ceph mount), this all did work just as expected.

When attempting to log in with just the keys, and then ctrl+C'ing out, the following gets added to /var/log/secure:

Aug  8 23:22:34 server sshd[16775]: Connection closed by 132.229.44.36 port 59610 [preauth]
Aug  8 23:22:51 server unix_chkpwd[16780]: password check failed for user (johndoe)
Aug  8 23:22:51 server sudo: pam_unix(sudo:auth): authentication failure; logname=johndoe uid=200104 euid=0 tty=/dev/pts/0 ruser=johndoe rhost=  user=johndoe
Aug  8 23:22:51 server sudo: johndoe : TTY=pts/0 ; PWD=/home/johndoe ; USER=root ; COMMAND=/bin/cat /var/log/secure
Aug  8 23:22:51 server sudo: pam_unix(sudo:session): session opened for user root by johndoe(uid=0)