1

I think I've got a dead system now, but I don't know how to recover.

I was updating from Buster to Bookworm with the standard method outlined here - https://wiki.debian.org/DebianUpgrade

After starting the upgrade, I had the error /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory and the upgrade stopped. Quickly searched for the solution and found this answer to resolve it, so started to execute that.

When trying to run sudo cp -av lib/x86_64-linux-gnu/* /lib/x86_64-linux-gnu/, I had a strange sudo error, something along the lines of cannot escalate, is your account locked? I don't know the exact error. I assumed a quick reboot would solve it, but this is where the problem started.

Once the computer was booted, I attempted to SSH into it and kept getting connection refused. I run this headless most of the time, so this was obviously a bad sign. I took it down from where it normally is and plugged it into a monitor to figure out what was going on.

The machine booted fine, but at the login prompt whenever I typed my username I'd get an immediate Login incorrect before even being prompted for the password. I tried root and also got Login incorrect.

I attempted to boot into debian recovery, and was met with the same error. No matter what I do, I can't log in to the system.

Here is a screenshot of the log before the login prompt.

boot log

I've attempted to boot into rescue mode from a usb drive, and I've been able to get a prompt on the system. Trying journalctl just gives me "No journal files were found."

The only thing I can find that might be of note is in /var/log/auth.log, which is;

PAM unable to dlopen(pam_unix.so): /lib/security/pam_unix.so: cannot open shared object file: No such file or directory
PAM adding faulty module: pam_unix.so
FAILED LOGIN (1) on `/dev/tty1` for `root`, Authentication Failure

Does anyone know how I can recover the system from this point? I've got absolutely no idea what to do. All I can access is the USB rescue mode prompt, so the actions I have are relatively limited.

Thanks

Improve this question
2
  • 1
    Buster and Bookworm are two stable releases apart (there was Bullseye in between), so if you tried upgrading directly, that may have caused the problem. Commented Jan 3, 2024 at 9:54
  • I did, that's probably a good call. Didn't occur to me that it would cause issues Commented Jan 4, 2024 at 14:28

2 Answers 2

Reset to default
3

Having just recovered from this state, this is what I did;

  • Boot using the Debian Live installer, and go into recovery.
  • Load the existing installation in recovery
  • Mount all relevant drives, in my case /var and /usr weren't mounted so that casued a few issues.
  • Following information here,

Run

cd /tmp  
apt download libcrypt1  
dpkg-deb -x libcrypt1_1%3a4.4.33-2_amd64.deb  .  
cp -av lib/x86_64-linux-gnu/* /lib/x86_64-linux-gnu/
apt --fix-broken install
  • Reboot, login now works since the broken state of PAM has been resolved
  • finish sudo apt full-upgrade

This has my system back in a fully usable, functional, and upgraded state. It seems that it broke libcrypt during the upgrade, crashed out of the upgrade because of that, and then broke login due to libcrypt being broken.

As a note, I also had an issue where the kernel didn't upgrade due to aufs-dkms, but after removing that it upgraded fine. I was still on v3.16, apparently that had been an issue for a while!

Improve this answer
5
  • 1
    Just had the exact same problem after a dist-upgrade failed. This solution worked for me. Thank you very much! Commented Feb 13, 2024 at 10:19
  • Glad it was able to help someone! Commented Feb 28, 2024 at 15:18
  • Sadly this isn't an option with a cloud based server. Commented Mar 21, 2024 at 5:48
  • Thank you so much for this, I did this to myself, but not even while attempting to upgrade per se. I was running an apt upgrade and for some reason I can't remember, I Ctrl+C'd it without thinking while it was in the middle of working, and I believe that was the culprit somehow, but this exact same issue was happening. I couldn't log in. This worked perfectly Commented Sep 21 at 17:14
  • OMG Thank you so much, you saved me! I had the exact same problem and your solution works like a charm. Thank you again for the very straight forward fix explanation! You're the king! Commented Oct 2 at 9:19
2

So, somehow your libcrypto.so doesn't work with the rest of your system. Maybe an interrupted upgrade? Maybe it was something manually patched and excluded? Hard to guess.

Anyway, libcrypt is involved in most system tools (it contains, amongst other things, password hashing routines), so all the executables that would allow you to authenticate are currently dysfunctional.

Anyways, the solution is usually to boot a Debian live USB stick, and reinstall the package containing libcrypto.so into your dysfunctional main system. This involves mounting the original system's root (say, on /target), and then doing something like

apt-get download libpam-modules libcrypt1
sudo dpkg -i --root=/target libpam-modules*.deb librypt1*.deb
Improve this answer
1
  • You are correct about the interrupted upgrade, I've just managed to recover. Thanks for the suggestion! I've posted another answer with my solution since I was thankfully able to get a command prompt in the main system via the recovery and didn't have to do it from another. Commented Jan 3, 2024 at 7:52

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.