This is documented in crypt(3)’s manpage, which you can find via shadow(5)’s manpage, or passwd(5)’s. Those links are appropriate for modern Linux-based systems; the description there is:
If salt is a character string starting with the characters "$id$" followed by a string optionally terminated by "$", then the result has the form:
$id$salt$encrypted
id identifies the encryption method used instead of DES and this then determines how the rest of the password string is interpreted. The following values of id are supported:
ID | Method ───────────────────────────────────────────────────────── 1 | MD5 2a | Blowfish (not in mainline glibc; added in some | Linux distributions) 5 | SHA-256 (since glibc 2.7) 6 | SHA-512 (since glibc 2.7)
So if a hashed password is stored in the above format, you can find the algorithm used by looking at the id; otherwise it’s crypt’s default DES algorithm.
Other platforms support other algorithms, so check the crypt manpage there. For example, OpenBSD’s crypt(3) supports DES (which is the default since Unix V7) and Blowfish, which it identifies using the id “2b”.