Automatic transformation of newlines in shell variable assignment

Question

This came out of one of my comments to this question regarding the use of bc in shell scripting. bc puts line breaks in large numbers, e.g.:

> num=$(echo 6^6^3 | bc)
> echo $num
12041208676482351082020900568572834033367326934574532243581212211450\ 20555710636789704085475234591191603986789604949502079328192358826561\ 895781636115334656050057189523456

But notice they aren't really line breaks in the variable -- or at least there are not if it is used unquoted. For example, in fooling around with more pipe in the assignment, e.g.:

num=$(echo 6^6^3 | bc | perl -pne 's/\\\n//g')

I realized that while there really is an \n in the bc output, checking echo $num > tmp.txt with hexdump shows the \n (ASCII 10) has definitely become a space (ASCII 32) in the variable assignment.

Or at least, in the output of unquoted $num >. Why is that?

As fedorqui points out, if you use quotes: echo "$num", you get newlines again. This is evident by examining the difference between echo $num > tmp.1 and echo "$num" > tmp.2 with hexdump; the former contains \ (backslash space) whereas the later contains \\n (backslash newline).

Answer 1

No idea why its there, but here's how to disable it with the GNU implementation of bc:

echo '6^6^3' | BC_LINE_LENGTH=0 bc

BC_LINE_LENGTH

This should be an integer specifying the number of characters in an output line for numbers. This includes the backslash and newline characters for long numbers. As an extension, the value of zero disables the multi-line feature. Any other value of this variable that is less than 3 sets the line length to 70.

Update:

I was confused about this question, I thought it was about the origins of the multi-line feature, it does seem like an odd one. Anyway the real answer is that if you do not quote the variable, the shell will do word splitting on it before this is passed to echo. Word splitting is the process where an expansion is split into 'words' depending on the contents of IFS, these 'words' then become different arguments. In the question example, this creates two arguments to echo, which echo then separates with a space (I knew this before Stephane commented, honest...).

To prevent this happening, just double quote the variable:

num=$(echo '6^6^3' | bc)
echo "$num"

Sometimes this is actually useful as a way to remove IFS characters from a variable (although printf %s is safer for arbitrary strings). Eg (in bash):

$ var=$'spaces:    newlines:\n\n\ntabs:\t\t\t end'

$ echo "$var"
spaces:    newlines:


tabs:            end
$ newvar="$(printf '%s ' $var)"
$ echo "$newvar"
spaces: newlines: tabs: end

Answer 2

echo puts a space between each two arguments. The shell considers the newline in $num just a word separator (just like space).

lines="a
b
c"
set -x
echo $lines   # several arguments to echo
echo "$lines" # one argument to echo

See this answer (by the OP himself) for a more detailed explanation.

Answer 3

Use tr to delete the line continuations and the newlines:

$ num=$(echo 6^6^3 | bc)
$ echo "$num"
12041208676482351082020900568572834033367326934574532243581212211450\
20555710636789704085475234591191603986789604949502079328192358826561\
895781636115334656050057189523456
$ num=$(echo "$num" | tr -d '\n\\')
$ echo "$num"
12041208676482351082020900568572834033367326934574532243581212211450205557106367897040854752345911916039867896049495020793281923588265618957816361153346560500571895 23456

Answer 4

In the bc man page, under expressions, it does explain the limit.

Since numbers are of arbitrary precision, some numbers may not be printable on a single output line. These long numbers will be split across lines using the "\" as the last character on a line. The maximum number of characters printed per line is 70.

Answer 5

As per Stephane Chazelas's comment, the issue not exactly that the space is interpreted as a newline. He's explained this elsewhere in relation to a (conceptual) "split+glob" operator, and although I don't totally follow the semantics of that, here's what's happening:

• The unquoted variable is split on whitespace by the shell and these are passed as arguments to echo. The splitting removes the whitespace. This may be a bit confusing since "backslash-newline" seems like an escape sequence, but it is NOT equivalent to x=\\\n, it's is equivalent to x=$'\\\012', \012 being octal for an actual newline -- as opposed to the escape sequence \\n, which results in the string containing a literal "\n" (backslash-n), which would not be split.

• echo then outputs its arguments separated by a space.

To reiterate:

str1=hello\\nworld
str2=$'hello\012world'

The shell will not split the first one unquoted since it doesn't actually contain any whitespace -- it contains the escape sequence \n, whereas in the second case it will be split unquoted on the actual newline character (which is whitespace).