Any privileged (UID=0) program can read /etc/shadow, retrieving both the encrypted password and the encryption algorithm for a given user.
The program uses the encryption algorithm to encrypt the password-to-be-tested. If the two encrypted passwords match, so do the two passwords, if they don't match, the passwords don't either, anand no information about the actual password is leaked.