-hmac takes the key as an argument (see manual), so your command asks for an HMAC using the key -hex. hexkey:... is taken as a filename, since it doesn't start with a dash, and so is the following -out.

To get the HMAC with a key given as a hex string, you'll need to use -mac hmac and -macopt hexkey:<key>. Note that using -hmac <key> and -mac hmac together doesn't work, and -macopt requires -mac hmac.

Test:

openssl dgst -sha256 -hmac abc <<< "message"
openssl dgst -sha256 -hmac abc -macopt hexkey:12345678 <<< "message"
openssl dgst -sha256 -mac hmac -macopt hexkey:616263 <<< "message"
perl -MDigest::HMAC=hmac_hex -MDigest::SHA=sha256 \
    -le 'print(hmac_hex("message\n", "abc", \&sha256))'

All give the hash 99592e56fcde028fb41882668b0cbfa0119116f9cf111d285f5cedb000cfc45a.

So, it seems you'd probably want

openssl dgst -sha256 -mac hmac -macopt hexkey:$(cat mykey.txt) -out hmac.txt /bin/ps

Since we're talking about cryptography, which is hard; and OpenSSL, which doesn't always have the most easy-to-use interfaces, I would suggest also verifying everything yourself, at least twice, instead of taking my word for it.

-hmac takes the key as an argument (see manual), so your command asks for an HMAC using the key -hex. hexkey:... is taken as a filename, since it doesn't start with a dash, and openssl doesn't take options after filenames, so the following -out is also a filename.

To get the HMAC with a key given as a hex string, you'll need to use -mac hmac and -macopt hexkey:<key>. Note that using -hmac <key> and -mac hmac together doesn't work, and -macopt requires -mac hmac.

Test:

openssl dgst -sha256 -hmac abc <<< "message"
openssl dgst -sha256 -hmac abc -macopt hexkey:12345678 <<< "message"
openssl dgst -sha256 -mac hmac -macopt hexkey:616263 <<< "message"
perl -MDigest::HMAC=hmac_hex -MDigest::SHA=sha256 \
    -le 'print(hmac_hex("message\n", "abc", \&sha256))'

All give the hash 99592e56fcde028fb41882668b0cbfa0119116f9cf111d285f5cedb000cfc45a which agrees with a random online HMAC calculator for message message\n, key abc or 616263 in hex. (Note the newline at the end of message here.)

So, it seems you'd probably want

openssl dgst -sha256 -mac hmac -macopt hexkey:$(cat mykey.txt) -out hmac.txt /bin/ps

Since we're talking about cryptography, which is hard; and OpenSSL, which doesn't always have the most easy-to-use interfaces, I would suggest also verifying everything yourself, at least twice, instead of taking my word for it.