How can I parallelize command sha256sum or other hashing commands?

For parallelizing across files you can use GNU Parallel:

parallel sha256sum ::: *

Parallelizing hashing for a single file can be done with certain hash functions using a Merkle tree. b3sum is such a tool.

Alternatively you could upgrade to a CPU which supports hardware SHA256 calculations like newer Intel or AMD CPUs. My Ryzen 3700X is blazingly fast at calculating SHA256 hash sums, a lot faster than for calculating MD5 which is a much simpler algo. Check for sha_ni in /proc/cpuinfo.

Alternatively you could use Blake3 which is a lot (almost 15 times) faster even when implemented purely in software:

xargs has a -P option for running multiple jobs in parallel. It's nowhere near as flexible as Ole Tange's parallel program, but it's fine for most simple parallelisation tasks.

For example:

find . -name '*.mp4' -print0 | xargs -0r -n 1 -P 0 openssl dgst -sha256

• -P 0 tells xargs to run as many jobs in parallel as possible (e.g. on my 32 thread Threadripper 1950x, it will run 32 openssl jobs in parallel)

• -n 1 tells xargs to run each job with only one argument. While 1 job is sub-optimal for openssl dgst which can process many filenames on a command line, you will almost always want to use the -n (or -L) option with xargs -P.

  Otherwise it will try to fit as many args onto each command line as it can - usually resulting in just one job unless you have many tens of thousands of arguments. On Linux, the command line length limit is typically 2 million characters, 2097152 (check with getconf ARG_MAX). That allows for a lot of filenames.

  The optimum is to count how many args you have and divide that by the number of jobs you want to run in parallel. e.g.

  t=32
  numfiles=$(find .//. -name '*.mp4' | grep -c //)
  let n=numfiles/t
  find . -name '*.mp4' -print0 | xargs -0r -n "$n" -P "$t" openssl dgst -sha256
  

Note: -P is not a standard POSIX option for xargs. Requires GNU or *BSD xargs. Maybe some other versions too.