3

After SFTP logins were no longer working with error 127 we found that the SSH connection created some output by a script installed in /etc/profile.d/.

Previous versions of sshd had an option available called "UseLogin" which by default was set to "no", thus did not execute stuff from profile. Now that option no longer seems to exist and the default behavior seems to have changed.

In sshd_config I now replaced

Subsystem  sftp  /usr/lib/ssh/sftp-server

by

Subsystem sftp  internal-sftp

This seems to work better (SFTP is working).

But as we all know: If you don't understand why it works, it isn't fixed.

Can someone explain? And maybe suggest a better "fix"?

Update SLES15.4 (openssh-server-8.4p1)

Improve this question
3
  • Please tell us a little more about your system: Which operating system (name, version) and which version of ssh and sftp are you running? Commented Jan 27, 2023 at 9:45
  • @sudodus updated Commented Jan 27, 2023 at 9:50
  • In my Ubuntu 22.04.x LTS I have openssh-server 1:8.9p1-3ubuntu0.1 and it works 'as usual' for me. It uses Subsystem sftp /usr/lib/ssh/sftp-server which used to work for you. I hope that someone with a system more similar to yours will see your question and help you find an answer. Commented Jan 27, 2023 at 10:09

2 Answers 2

Reset to default
5

SFTP uses SSH as transport. Any SFTP client expects the SSH server to establish communication with an SFTP server (like sftp-server).

At least with OpenSSH, when an SSH server is told to run something, it uses the target user's shell for this.

One can define a subsystem (it may be a custom subsystem) by adding Subsystem … entry in sshd_config. Even then the supplied command will be executed in the user's shell. This happened to you with the standard sftp subsystem specified as /usr/lib/ssh/sftp-server.

If the shell (or anything really) prints some "garbage" when your SFTP client expects to talk to an SFTP server, it's outside of the SFTP protocol and thus the communication breaks.

As long as the SSH server uses the user's shell, no option can totally reliably make everything work. This is because in general:

  1. The user's shell may be anything.
  2. Even sane shells source some files. Some shells may be told not to, but there is no portable option for this and there is no way for a client to tell the SSH server to use a custom option when invoking a shell.
  3. The sourced files may print something; or they may run something that prints something.

The only way to avoid the user's shell is to use a subsystem handled internally by the SSH server. AFAIK for now the only internal subsystem in sshd from OpenSSH is internal-sftp.

internal-sftp solved your problem because it does not rely on a shell.

An alternative fix is to make sure nothing but SFTP server uses the standard streams provided by the SSH server. This solution includes silencing the user's shell and anything the shell starts before it runs the actual requested command like /usr/lib/ssh/sftp-server. A person with root access on the server or the user himself/herself may easily break this.

Some interesting cases, for comparison:

Improve this answer
1
  • Your answer doesn't mention the (apparently no longer present) UseLogin option. But it is very well written and addresses the question of better understanding what is happening, which is why I'm accepting it. Anyone: For the actual and practical resolution, please see my own answer (probably below). Commented Jan 28, 2023 at 18:05
2

Never mind.

Someone dropped LD_LIBRARY_PATH=/some/path into /etc/profile.d/xxx, with /some/path containing outdated libs like libcrypto to make some ancient stuff work. Suffice to say I'm not amused. This is also what broke sftp-server

/usr/lib/ssh/sftp-server: symbol lookup error: /usr/lib/ssh/sftp-server: undefined symbol: EVP_KDF_CTX_new_id, version OPENSSL_1_1_1d

UPDATE It is not yet clear if fixing the above would have made sftp-server work again as there is still the output created from profile.d. I haven't tried. It probably would only have fixed error/exit code 127 but then generated a different error.

As internal-sftp uses neither the bad libs nor creates a login shell (for details see answer by @kamil-maciorowski), it is probably the only way forward. Except for tweaking the script in profile.d to only execute for certain uids (like not for uid 0) and thus not generate any output in certain cases.

Improve this answer
2
  • 3
    The question states "we found that the SSH connection created some output by a script" and my answer addresses the generic problem of noisy startup scripts. Your answer reveals that sftp-server couldn't work due to LD_LIBRARY_PATH, so even if everything was silent, SFTP wouldn't work. For a moment I considered deleting my answer, as the answer could not help you by itself. Now I think it's good to have my answer that explains a relatively common issue, along your answer that reveals the actual, somewhat unexpected culprit. Commented Jan 27, 2023 at 11:20
  • @KamilMaciorowski you should definitely leave your answer. My initial deduction was not entirely correct. Error 127 means the binary cannot be found, or as in this case apparently cannot be executed. I haven't even tried yet if fixing the problems with dynamic linker would make sftp-server work (while the output from profile is still present). Commented Jan 28, 2023 at 12:15

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.