and tried to SSH into the host machine again, expecting it to fail (since the public key is missing), but it still works! How come?

Once you have sent your public key to the remote server (and the remote server has stored your public key in its authorized_keys file), you don't need your public key file anymore. All that you need to authenticate is your private key.

Of course, it's good practice to keep the public key file. But since a RSA private key contains the public key, you can always extract the public key file from the private key file.

For detail, this is the content of a RSA private key:

-----BEGIN RSA PRIVATE KEY-----
RSAPrivateKey ::= SEQUENCE {
  version           Version,
  modulus           INTEGER,  -- n
  publicExponent    INTEGER,  -- e
  privateExponent   INTEGER,  -- d
  prime1            INTEGER,  -- p
  prime2            INTEGER,  -- q
  exponent1         INTEGER,  -- d mod (p-1)
  exponent2         INTEGER,  -- d mod (q-1)
  coefficient       INTEGER,  -- (inverse of q) mod p
  otherPrimeInfos   OtherPrimeInfos OPTIONAL
}
-----END RSA PRIVATE KEY-----

and this is the content of a RSA public key:

-----BEGIN RSA PUBLIC KEY-----
RSAPublicKey ::= SEQUENCE {
    modulus           INTEGER,  -- n
    publicExponent    INTEGER   -- e
}
-----END RSA PUBLIC KEY-----   

and tried to SSH into the host machine again, expecting it to fail (since the public key is missing), but it still works! How come?

Once you have sent your public key to the remote server (and the remote server has stored your public key in its authorized_keys file), you don't need your public key file anymore. All that you need to authenticate is your private key.

Of course, it's good practice to keep the public key file. But since a RSA private key contains the public key, you can always extract the public key file from the private key file.

For detail, this is the content of a RSA private key:

-----BEGIN RSA PRIVATE KEY-----
RSAPrivateKey ::= SEQUENCE {
  version           Version,
  modulus           INTEGER,  -- n
  publicExponent    INTEGER,  -- e
  privateExponent   INTEGER,  -- d
  prime1            INTEGER,  -- p
  prime2            INTEGER,  -- q
  exponent1         INTEGER,  -- d mod (p-1)
  exponent2         INTEGER,  -- d mod (q-1)
  coefficient       INTEGER,  -- (inverse of q) mod p
  otherPrimeInfos   OtherPrimeInfos OPTIONAL
}
-----END RSA PRIVATE KEY-----

and this is the content of a RSA public key:

-----BEGIN RSA PUBLIC KEY-----
RSAPublicKey ::= SEQUENCE {
    modulus           INTEGER,  -- n
    publicExponent    INTEGER   -- e
}
-----END RSA PUBLIC KEY-----   

(Source: RFC 3447.)