how to know if we reached the max value of fs.file-max

This

[[ `lsof | wc -l` -gt 100000 ]] && echo "please increase the fs.file-max"

will be misleading. See reference below.

Based on RHEL 5 Tuning, the output of cat /proc/sys/fs/file-nr can be used. We can compare the first and last fields of the output. The first field shows total allocated file handles and the last field shows the maximum file handles that can be allocated. So setup a check so that once, say 90% is reached, a warning is displayed:

[[ $(echo "$(cat /proc/sys/fs/file-nr | cut -f1) / $(cat /proc/sys/fs/file-nr | cut -f3) * 100" | bc -l) -ge 90 ]] \
  && echo "90% reached. Please increase the fs.file-max"

Some difference between lsof and file-nr output:

The file-max kernel parameter refers to open file descriptors, and file-nr gives us the current number of open file descriptors. But lsof lists all open files, including files which are not using file descriptors – such as current working directories, memory mapped library files, and executable text files.

See Reference