Read line by line from a variable in shell scripting

Question

I have a script variable which is multi-line. How do i traverse this variable to read it line by line and process each line the way I want?

There is a similar question at unix.stackexchange.com/questions/9784/…; I have written an answer there. — Binarus
– Binarus, Commented Aug 20, 2020 at 19:58

TomRoche · Accepted Answer · 2018-01-08 09:27:04Z

37

Consider the following multi-line variable

x=$(echo -e "a\nb\nc d e")

and a simple process for each line: just echo it with a prefix=LINE: and with single quotes around the line. Either of the following codes will satisfy that requirement:

while read line; do echo "LINE: '${line}'"; done <<< "$x"

or

while read line; do echo "LINE: '${line}'"; done < <(echo "$x")

Neither creates a subshell (so you can, e.g., set variables in the loop and access them outside of it), and both output

LINE: 'a'
LINE: 'b'
LINE: 'c d e'

But suppose instead you have

x=$(echo -e "a \n b\nc d e")
# note--------^--^

and that leading and trailing whitespace matter for your application (e.g., parsing Git porcelain). Both the above codes will give exactly the same output for the latter variable/data as for the former, which is not what you want. To preserve leading and trailing whitespace, replace while read line with while IFS= read -r line . I.e., either of the following codes

while IFS= read -r line; do echo "LINE: '${line}'"; done <<< "$x"

or

while IFS= read -r line; do echo "LINE: '${line}'"; done < <(echo "$x")

will produce

LINE: 'a '
LINE: ' b'
LINE: 'c d e'

See Greg Wooledge's excellent Bash FAQ for details.

edited Jan 8, 2018 at 9:27

TomRoche

1,5341 gold badge17 silver badges26 bronze badges

answered Aug 10, 2012 at 13:02

Karoly Horvath

96.7k11 gold badges123 silver badges181 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Senthil Kumar Over a year ago

this is a perfect solution, just curious, why not just x="a\nb\nc d e"?

chepner Over a year ago

@SenthilKumar, that string contains 10 charactes: 'a', '\', 'n', 'b', etc. The "\n" are not special inside double quotes. In bash, one could write $"a\nb\nc d e" to have real newlines embedded in the string. The "-e" flag tells echo to process "\n" as a new line, not two indvidual characters.

Karoly Horvath Over a year ago

@chepner: thx, $ was new to me, but it works with single quotes $'a\nb\nc d e'

chepner Over a year ago

@Karoly: my mistake, I used the wrong quotes. Double quotes are for gettext translations; single quotes are for backslash expansions.

Binarus Over a year ago

It should be mentioned that this solution only works because echo adds a trailing newline; in other words, it alters the variable to parse. This might or might not be acceptable for the OP. Without the trailing newline, read would completely miss the last line. For details, see unix.stackexchange.com/a/9789

Seth McCauley · Accepted Answer · 2020-10-18 18:44:46Z

5

Although I typically use "while read" for processing multi-line variables, I recently had an instance where it removed the leading space from each line in a file. Using this instead fixed my issue:

printf '%s\n' "$var" | while IFS= read -r line
do
   echo "$line"
done

Code taken from this Unix Stack Exchange answer.

Edit: updated to fix last line issue as suggested by Nicolae Iotu

edited Oct 18, 2020 at 18:44

answered Mar 20, 2014 at 22:36

Seth McCauley

1,02314 silver badges25 bronze badges

2 Comments

Nicolae Iotu Over a year ago

Nice solution, but you will need to end printf with a new line to make it process the last line: printf '%s\n' "$var" ...

Seth McCauley Over a year ago

Good point, thank you. I have updated my answer with the changes you suggested.

Collectives™ on Stack Overflow

Read line by line from a variable in shell scripting

2 Answers 2

5 Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Related