How to make bash built-in "read" ignore commented or empty lines?

Question

(For simplicity I'll assume the file to read is the first argument - $1.)

I can do what I want externally with:

tempfile=$(mktemp)
awk '/^#/ {next}; NF == 0 {next}; {print}' "$1" > $tempfile
while read var1 var2 var3 var4 < $tempfile; do
  # stuff with var1, etc.
done

However, it seems absurd to need to call awk every time I parse the config file. Is there a way to make read ignore commented or whitespace-only lines in a file, without external binaries/potential performance issues?

---

Answers so far are quite helpful! To clarify, I don't want to use a temp file, but I do want to read the config from a file, not from standard in. I'm well aware that I can use an input redirection when I call the script, but for various reasons that won't work in my circumstance.

I want to softcode the input to read from, e.g.:

configfile="/opt/myconfigfile.txt"
[ $# -gt 0 ] && [ -r "$1" ] && configfile="$1"

while read var1 var2 var3 var4 < "$configfile" ; do
  ...

But when I try this, it just reads the first line of configfile over and over until I kill the process.

Maybe this should be its own question...but it's probably a single line change from what I'm doing. Where's my error?

Answer 1

You don't need a tempfile to do this, and sed (or awk) are far more flexible in comment processing than a shell case statement.

For example:

configfile='/opt/myconfigfile.txt'
[ $# -gt 0 ] && [ -r "$1" ] && configfile="$1"

sed -e 's/[[:space:]]*#.*// ; /^[[:space:]]*$/d' "$configfile" |
    while read var1 var2 var3 var4; do
      # stuff with var1, etc.
    done

# Note: var1 etc are not available to the script at this
# point. They are only available in the sub-shell running
# the while loop, and go away when that sub-shell ends.

This strips comments (with or without leading whitespace) and deletes empty lines from the input before piping it into the while loop. It handles comments on lines by themselves and comments appended to the end of the line:

# full-line comment
# var1 var2 var3 var4
abc 123 xyz def # comment here

Calling sed or awk for tasks like this isn't "absurd", it's perfectly normal. That's what these tools are for. As for performance, I'd bet that in anything but very tiny input files, the sed version would be much faster. Piping to sed has some startup overhead but runs very fast, while shell is slow.

---

Update 2022-05-03:

Note that the variables (var1, var2, var3, etc) which are set in the while read loop will "go out of scope" when the while loop ends. The can only be used inside that while loop. The while loop is being run in a sub-shell because the config file is being piped into it. When that sub-shell dies, its environment goes with it - and a child process can not change the environment of its parent process.

If you want the variables to retain their values after the while loop, you need to avoid using a pipe. For example, use input redirection (<) and process substitution (<(...)):

while read var1 var2 var3 var4; do
  # stuff with var1, etc.
done < <(sed -e 's/[[:space:]]*#.*// ; /^[[:space:]]*$/d' "$configfile")

# remainder of script can use var1 etc if and as needed.

With this process substitution version, the while loop runs in the parent shell and the sed script is run as a child process (with its output redirected into the while loop). sed and its environment goes away when it finished, while the shell running the while loop retains the variables created/changed by the loop.

Answer 2

This works because read breaks everything on whitespace (IFS), so if var1 is empty or starts with '#', then skip it.

while read var1 var2 var3 var4; do
   case $var1 in
       ''|\#*) continue ;;         # skip blank lines and lines starting with #
   esac
   echo "var1: '$var1'"
   # stuff with var1, etc.
done < "${1:-default_config_file}"

Then input has to be redirected into the loop instead of to the while command list. The "${1:-default_config_file}" expands to the first command line parameter if not empty, otherwise expands to default_config_file you can also use variable expansion, etc in the default value string.

Because you're interested in minimizing pre-processing, I think this is equivalent, but also removes all comments:

while read line; do
    echo "${line%%#*}" | {
        read var1 var2 var3 var4
        [ -z "$var1" ] && continue
        # stuff with var1, etc.
        for i in 1 2 3 4; do eval echo "\"var$i: \$var$i\""; done  #debug only!
    }
done < "${1:-default_config_file}"

This uses the shell parameter expansion substring processing feature. ${line%%#*} expands to the original value of line except the first # and everything after it are removed. Load that into var1-4 and continue as usual. The test to continue shortens because we now only need to check for empty string, not #.

Answer 3

You can do that without create temp file. The grep command will filter empty and commented lines.

while read var1 var2 var3; do
    echo $var1
    echo $var2
    echo $var3
    echo "etc..."
done < <(grep -v "^#\|^$" /opt/myconfigfile.txt)