Iterate through multiline string line by line

Question

I would like to process a multiline string and iterate it line by line, in a POSIX shell (/bin/sh) on a BSD platform. Bash is not included in the base BSD-distribution and has a GPL license - so I am trying to make it universally work with /bin/sh instead.

I found a solution using a pipe, however in the regular /bin/sh shell, these a processed in a separate process, meaning the following does not work:

MULTILINE="`cat ${SOMEFILE}`"
SOMEVAR="original value"

echo "${MULTILINE}" | while IFS= read -r SINGLELINE
do
 SOMEVAR="updated value"
 echo "this is a single line: ${SINGLELINE}"
 echo "SOMEVAR is now: ${SOMEVAR}"
done

echo "Final SOMEVAR is unchanged: ${SOMEVAR}"

In the above example, it accomplishes what I want, except for the fact that changes to variables such as ${SOMEVAR} are not accessible outside the while loop.

My question: how can I accomplish something like the above without this restriction? Note that many solutions require Bash, whereas I am using the standard POSIX-shell /bin/sh.

Answer 1

You could use a here document:

while IFS= read -r SINGLELINE
do
  SOMEVAR="updated value"
  printf '%s\n' "this is a single line: ${SINGLELINE}"
  printf '%s\n' "SOMEVAR is now: ${SOMEVAR}"
done << EOF
$MULTILINE
EOF
printf '%s\n' "Final SOMEVAL is still $SOMEVAR"

Depending on the sh implementation, here-documents are implemented either as a deleted temporary file where the shell has stored the expansion of the variable followed by newline beforehand, or a pipe to which the shell feeds the expansion of the variable followed by newline. But in either case, except in the original Bourne shell (a shell that is no longer in use these days and is not a POSIX compliant shell), the command being redirected is not run in a subshell (as POSIX requires).

or you could use split+glob:

IFS='
' # split on newline only
set -o noglob
for SINGLELINE in $MULTILINE
do
  SOMEVAR="updated value"
  printf '%s\n' "this is a single line: ${SINGLELINE}"
  printf '%s\n' "SOMEVAR is now: ${SOMEVAR}"
done
printf '%s\n' "Final SOMEVAL is still $SOMEVAR"

But beware it skips empty lines.

Answer 2

You would read directly from the file without the pipeline. This avoids running the while loop in a subshell, which allows you to see the changed value of $SOMEVALUE after the loop.

SOMEVAR="original value"

while IFS= read -r SINGLELINE
do
    SOMEVAR="updated value"
    printf 'this is a single line: %s\n' "$SINGLELINE"
    printf 'SOMEVAR is now: %s\n' "$SOMEVAR"
done <"$SOMEFILE"

printf 'Final SOMEVAR is: %s\n' "$SOMEVAR"

If you insist on having your $MULTILINE variable, then write that to a file and read it from there:

tmpfile=$(mktemp)
printf '%s\n' "$MULTILINE" >"$tmpfile"

while ...; do
   ...
done <"$tmpfile"
rm "$tmpfile"

Also related:

• Why is my variable local in one 'while read' loop, but not in another seemingly similar loop?

An answer to the above linked question also suggests writing your program in such a way that all uses of $SOMEVAR occurs within the subshell at the end of the pipeline:

MULTILINE=$(cat "$SOMEFILE")
SOMEVAR="original value"

printf '%s\n' "$MULTILINE" | {
    while IFS= read -r SINGLELINE
    do
        SOMEVAR="updated value"
        printf 'this is a single line: %s\n' "$SINGLELINE"
        printf 'SOMEVAR is now: %s\n' "$SOMEVAR"
    done

    printf 'Final SOMEVAR is: %s\n' "$SOMEVAR"
}

Also possibly related:

• Why is using a shell loop to process text considered bad practice?

Other questions that may be of interest:

• Why is printf better than echo?
• Are there naming conventions for variables in shell scripts?

Answer 3

It works for me :

$ cat bin/test
#! /bin/sh
SOMEFILE=$1
MULTILINE="`cat ${SOMEFILE}`"
SOMEVAR="blah"

echo "${MULTILINE}" | while IFS= read -r SINGLELINE
do
 echo "this is a single line: ${SINGLELINE}"
 echo "but accessing this var fails: ${SOMEVAR}"
done

and

$ bin/test bin/test
this is a single line: #! /bin/sh
but accessing this var fails: blah
this is a single line: SOMEFILE=$1
but accessing this var fails: blah
this is a single line: MULTILINE="`cat ${SOMEFILE}`"
but accessing this var fails: blah
this is a single line: SOMEVAR="blah"
but accessing this var fails: blah
this is a single line:
but accessing this var fails: blah
this is a single line: echo "${MULTILINE}" | while IFS= read -r SINGLELINE
but accessing this var fails: blah
this is a single line: do
but accessing this var fails: blah
this is a single line:  echo "this is a single line: ${SINGLELINE}"
but accessing this var fails: blah
this is a single line:  echo "but accessing this var fails: ${SOMEVAR}"
but accessing this var fails: blah
this is a single line: done
but accessing this var fails: blah