Return to Answer

OP and I worked through this; see chat for details. First, OP changed

/sbin/sulogin

in /etc/init/mountall-shell.conf to

/usr/bin/ltrace -S -f -o /root/sulogin-ltrace.log /bin/sulogin

The log indicates that the segfault occurs in the following code, where crypt is returning NULL.

if ((p = getpasswd(pwd->pw_passwd)) == NULL) break;
if (pwd->pw_passwd[0] == 0 ||
    strcmp(crypt(p, pwd->pw_passwd), pwd->pw_passwd) == 0)
        sushell(pwd);

Next question is, what's causing crypt to return NULL?

OP said that the shadow entry for root is root:x:16273:0:99999:7:::. The encrypted password is ! in a stock Ubuntu 14.04 system; they changed it to x and this is the first time since then that OP has had to use single-user mode.

sulogin has its own interpretation of special encrypted passwords. If it sees * or !, it lets the user in with no password. Anything else, it does some validity checking, but "x" sails right through, yet crypt doesn't like it and returns NULL.

OP is going to file a bug report for sysvinit-utils; sulogin ought to handle a NULL return from crypt more gracefully.

OP and I worked through this; see comments & chat for details. First, to find the problem process and location, this line in /etc/init/mountall-shell.conf

/sbin/sulogin

was changed to

/usr/bin/ltrace -S -f -o /root/sulogin-ltrace.log /bin/sulogin

Excerpt from log:

837 crypt("password", "x") = nil
837 strcmp(nil, "x" <no return ...>
837 --- SIGSEGV (Segmentation fault) 

The log indicates that the segfault occurs in the following code in sulogin, where crypt is returning NULL.

if ((p = getpasswd(pwd->pw_passwd)) == NULL) break;
if (pwd->pw_passwd[0] == 0 ||
    strcmp(crypt(p, pwd->pw_passwd), pwd->pw_passwd) == 0)
        sushell(pwd);

Next question is, what's causing crypt to return NULL?

OP confirmed that the encrypted password really was x; the shadow entry for root was root:x:16273:0:99999:7:::. In a stock Ubuntu 14.04, root's encrypted password is !; OP had changed it to x awhile ago and this is the first time since then that he's had to use single-user mode.

sulogin has its own interpretation of special encrypted passwords. If it sees * or !, it lets the user in with no password. Anything else, it does some validity checking, but x sails right through, yet crypt doesn't like it (salt not long enough?) and returns NULL.

OP is going to file a bug report for sysvinit-utils; sulogin ought to handle a NULL return from crypt more gracefully.

OP and I worked through this; see chat for details. First, OP changed

/sbin/sulogin

in /etc/init/mountall-shell.conf to

/usr/bin/ltrace -S -f -o /root/sulogin-ltrace.log /bin/sulogin

The log indicates that the segfault occurs in the following code, where crypt is returning NULL.

if ((p = getpasswd(pwd->pw_passwd)) == NULL) break;
if (pwd->pw_passwd[0] == 0 ||
    strcmp(crypt(p, pwd->pw_passwd), pwd->pw_passwd) == 0)
        sushell(pwd);

Next question is, what's causing crypt to return NULL?

OP said that the shadow entry for root is root:x:16273:0:99999:7:::.

sulogin has its own interpretation of special encrypted passwords. If it sees "*" or "!", it lets the user in with no password. Anything else, it does some validity checking, but "x" sails right through, yet crypt doesn't like it and returns NULL.

OP is going to file a bug report for sysvinit-utils; sulogin ought to handle a NULL return from crypt more gracefully.

OP and I worked through this; see chat for details. First, OP changed

/sbin/sulogin

in /etc/init/mountall-shell.conf to

/usr/bin/ltrace -S -f -o /root/sulogin-ltrace.log /bin/sulogin

The log indicates that the segfault occurs in the following code, where crypt is returning NULL.

if ((p = getpasswd(pwd->pw_passwd)) == NULL) break;
if (pwd->pw_passwd[0] == 0 ||
    strcmp(crypt(p, pwd->pw_passwd), pwd->pw_passwd) == 0)
        sushell(pwd);

Next question is, what's causing crypt to return NULL?

OP said that the shadow entry for root is root:x:16273:0:99999:7:::. The encrypted password is ! in a stock Ubuntu 14.04 system; they changed it to x and this is the first time since then that OP has had to use single-user mode.

sulogin has its own interpretation of special encrypted passwords. If it sees * or !, it lets the user in with no password. Anything else, it does some validity checking, but "x" sails right through, yet crypt doesn't like it and returns NULL.

OP is going to file a bug report for sysvinit-utils; sulogin ought to handle a NULL return from crypt more gracefully.

That comes from the /etc/init/mountall-shell.conf file:

echo "Filesystem check or mount failed."
echo "A maintenance shell will now be started."
echo "CONTROL-D will terminate this shell and continue booting after re-trying"
echo "filesystems.  Any further errors will be ignored"
;;
...
/sbin/sulogin

So it's likely sulogin or one of the processes it execs/forks. Try temporarily mounting the root filesystem on a working VM (or mount it after booting from the live CD), and replace that line with this, which will trace all system calls:

/usr/bin/strace -f -o /root/sulogin-strace.log /bin/sulogin

and see what's in the log.

An alternative, which will trace shared library calls as well:

/usr/bin/ltrace -S -f -o /root/sulogin-ltrace.log /bin/sulogin

Do you know what the problem was that caused fsck or mount to fail? On Ubuntu, the console would show something like this:

The disk drive for /path/to/mountpoint is not ready or not present.
keys:Continue to wait, or Press S to skip mounting or M for manual recovery

or

Serious errors were found while checking the disk drive for /path/to/mountpoint.
keys:Press I to ignore, S to skip mounting, or M for manual recovery

If /path/to/mountpoint contains a shared library or a file that a PAM uses, that could cause sulogin to malfunction.

OP and I worked through this; see chat for details. First, OP changed

/sbin/sulogin

in /etc/init/mountall-shell.conf to

/usr/bin/ltrace -S -f -o /root/sulogin-ltrace.log /bin/sulogin

The log indicates that the segfault occurs in the following code, where crypt is returning NULL.

if ((p = getpasswd(pwd->pw_passwd)) == NULL) break;
if (pwd->pw_passwd[0] == 0 ||
    strcmp(crypt(p, pwd->pw_passwd), pwd->pw_passwd) == 0)
        sushell(pwd);

Next question is, what's causing crypt to return NULL?

OP said that the shadow entry for root is root:x:16273:0:99999:7:::.

sulogin has its own interpretation of special encrypted passwords. If it sees "*" or "!", it lets the user in with no password. Anything else, it does some validity checking, but "x" sails right through, yet crypt doesn't like it and returns NULL.

OP is going to file a bug report for sysvinit-utils; sulogin ought to handle a NULL return from crypt more gracefully.