132

Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:

#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output

if [ -z "$OUTPUT" ]
then
        echo "Status WARN: No messages from SMcli"
        exit $STATE_WARNING
else
        echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
        do
                if [ "$STATUS" != "Optimal" ]
                then
                        echo "CRIT: $NAME - $STATUS"
                        echo $((++XCODE))
                else
                        echo "OK: $NAME - $STATUS"
                fi
        done
fi

echo $XCODE

I've tried using the following statement instead of the ++XCODE method

XCODE=`expr $XCODE + 1`

and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.

5
  • 1
    Where do you initialize XCODE to something that can be incremented? Commented Sep 23, 2008 at 21:57
  • I've tried to throw an "XCODE=0" at the top of the code, outside of the while statement Commented Sep 23, 2008 at 21:59
  • Without the cruft, it works for me. #!/bin/bash for i in 1 2 3 4 5; do echo $((++XCODE)) done echo "fin:" $XCODE I think your problem has nothing to do with variable scoping and everything to do with what's happening in the while. Commented Sep 23, 2008 at 22:08
  • Agreed.. it seems like it has to do with the "while read" loop? Commented Sep 23, 2008 at 22:11
  • 1
    There's a Bash FAQ about this: mywiki.wooledge.org/BashFAQ/024 Commented Jul 5, 2019 at 13:37

7 Answers 7

170

Because you're piping into the while loop, a sub-shell is created to run the while loop.

Now this child process has its own copy of the environment and can't pass any variables back to its parent (as in any unix process).

Therefore you'll need to restructure so that you're not piping into the loop. Alternatively you could run in a function, for example, and echo the value you want returned from the sub-process.

http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL

Sign up to request clarification or add additional context in comments.

3 Comments

this just answered so many of the seemingly random issues i was running into with bash scripting.
This perfect answer upsets me so much and explains a really weird behaviour in our CI system.
This great answer would be improved with an example for the OP: while read NAME IP1 IP2 STATUS; do stuff-in-body-of-while-loop; done < <(echo "name1 ip ip status"), replacing the echo statement in that process substitution with the "[other] command with multi line output" mentioned in the original question. Pro Tip: ShellCheck will advise you to use read -r to avoid corrupting the input if it happens to contain backslashes.
127

The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.

Your specific example can be solved by rewriting the pipe to

while ... do ... done <<< "$OUTPUT"

or perhaps

while ... do ... done < <(echo "$OUTPUT")

5 Comments

For those who are looking on at this confused as to what the whole <() syntax is (like I was), it's called "Process Substitution", and the specific usage detailed above can be seen here: mywiki.wooledge.org/ProcessSubstitution
Process Substitution is something everyone should be using regularly! It is super useful. I do something like vimdiff <(grep WARN log.1 | sort | uniq) <(grep WARN log.2 | sort | uniq) every day. Consider that you can use multiple at once and treat them like files... POSSIBILITIES!
I had this exact issue, and refactoring the pipline to save the first stage output and write like this was trivially simple. Thank you so much for saving me hours of research and experimentation.
Or, to extend @BrunoBronosky 's comment, the way I often use process substitution is to diff local and remote files, e.g. diff -u <( ssh remote.example.com "cat file.txt") file.txt
fails for me git ls-remote --tags origin | while read -r line; do gist.github.com/kiquenet/0cca459b9b1a9cbb5578689b49a16739
13

This should work as well (because echo and while are in same subshell):

#!/bin/bash
cat /tmp/randomFile | (while read line
do
    LINE="$LINE $line"
done && echo $LINE )

Comments

4

I got around this when I was making my own little du:

ls -l | sed '/total/d ; s/  */\t/g' | cut -f 5 | 
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )

The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.

Comments

3

Another option is to output the results into a file from the subshell and then read it in the parent shell. something like

#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
    LINE="$LINE $line"
    echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)

Comments

3

One more option:

#!/bin/bash
cat /some/file | while read line
do
  var="abc"
  echo $var | xsel -i -p  # redirect stdin to the X primary selection
done
var=$(xsel -o -p)  # redirect back to stdout
echo $var

EDIT: Here, xsel is a requirement (install it). Alternatively, you can use xclip: xclip -i -selection clipboard instead of xsel -i -p

2 Comments

I get an error: ./scraper.sh: line 111: xsel: command not found ./scraper.sh: line 114: xsel: command not found
@3kstc obviously, install xsel. Also, you can use xclip, but its usage a little bit different. Main point here: 1st you put the output into a clipboard (3 of them in linux), 2nd - you grab it from there and send to stdout.
2
 #!/bin/bash
 OUTPUT="name1 ip ip status"
+export XCODE=0;
 if [ -z "$OUTPUT" ]
----

                     echo "CRIT: $NAME - $STATUS"
-                    echo $((++XCODE))
+                    export XCODE=$(( $XCODE + 1 ))
             else

echo $XCODE

see if those changes help

1 Comment

When doing this, I now get a "0" to print for the last echo statement. however I expect the value to be 1 not zero. Also, why the use of export? I assume that forces it into the environment?

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.