2

While debugging a script, I finally found the cause, but I don't understand the why.

with an example tab delimited text file containing:

$ cat list1.txt
123 Fake St Miami   FL
456 None Rd San Francisco CA
789 Nowhere Dr  Denver  CO

initially created with vi, file identifies it as

$ file list1.txt
list1.txt: ASCII text

Running this one liner (edit: changed post to multiline for readability) to read each line into 3 variables, the printed order is as specified in the script. Note that the concatenation of strings is this convoluted as I was trying to be very explicit about the order when debugging:

$ while IFS="     " read -r addr1 city state
do
  data0="'"
  data1='companyName=&'
  data2="address1=$addr1"
  data3='&city='
  data4="$city"
  data5='&state='
  data6="$state"
  data7='&urbanCode=&zip='
  data8="'"
  data=${data0}${data1}${data2}${data3}${data4}${data5}${data6}${data7}${data8}
  echo "$data"
done < list1.txt

'companyName=&address1=123&city=Fake&state=St   Miami   FL&urbanCode=&zip='
'companyName=&address1=456&city=None&state=Rd   San Francisco CA&urbanCode=&zip='
'companyName=&address1=789&city=Nowhere&state=Dr    Denver  CO&urbanCode=&zip='

Changing file list1.txt DOS format results in reordered output

$ unix2dos list1.txt
unix2dos: converting file list1.txt to DOS format...
$ file list1.txt
list1.txt: ASCII text, with CRLF line terminators
$ while IFS="     " read -r addr1 city state; do  data0="'";  data1='companyName=&';  data2="address1=$addr1";  data3='&city=';  data4="$city";  data5='&state=';  data6="$state";  data7='&urbanCode=&zip=';  data8="'";  data=${data0}${data1}${data2}${data3}${data4}${data5}${data6}${data7}${data8};  echo "$data"; done < list1.txt
&urbanCode=&zip='ress1=123&city=Fake&state=St   Miami   FL
&urbanCode=&zip='ress1=456&city=None&state=Rd   San Francisco CA
&urbanCode=&zip='ress1=789&city=Nowhere&state=Dr    Denver  CO

Why is this happening? GNU bash, version 3.2.57

Improve this question
2
  • 1
    For readability, especially when presenting code for others to read and understand, don't be afraid of expanding your one liner into a code block Commented Dec 29, 2022 at 22:13
  • 4
    TBH I stopped bothering to read after the second horizontal scroll and still no end in sight Commented Dec 29, 2022 at 22:14

1 Answer 1

Reset to default
6

When reading a line from the DOS-formatted text file into the three variables, the last one, state, will have a carriage-return character at the end. This is because DOS text files denote "newline" using the character sequence CR+LF (carriage-return followed by line-feed). Unix text files only use line-feed for newline, and the carriage-return of the DOS text file is treated as any other character at the end of the line.

When outputting $state to the teminal, this carriage-return character causes the position of the output to jump to the start of the line (this is the purpose of the carriage-return, to return the "carriage" (of the line printer, once upon a time) to the start of the line) and the following string to be placed at the very start of the line, over-writing any text previously outputted at these positions in the terminal.

So you get, for the first line,

'companyName=&address1=123&city=Fake&state=

followed by the value of $state, St Miami FL, and then the cursor is warped back to the start of the line, where

&urbanCode=&zip='

is outputted, overwriting the first part of the line, resulting in the weird-looking

&urbanCode=&zip='ress1=123&city=Fake&state=St   Miami   FL
Improve this answer
6
  • This would make perfect sense, except that the first line of output does not follow that pattern Commented Dec 29, 2022 at 22:54
  • @MJohnson why not? It seems just like the other lines. Commented Dec 29, 2022 at 23:04
  • @aviro This explanation for the DOS formatted file shows that the first line starts with companyName (which is the desired result), but the actual test resulted in the first (and all) lines beginning with &urbanCode Commented Dec 30, 2022 at 0:18
  • 2
    @MJohnson In my answer I say that first, 'companyName=&address1=123&city=Fake&state= is printed, followed by St Miami FL. Then the cursor goes back to the start of the line, and &urbanCode=&zip=' overwrites what was previously at the start of the line. This is exactly what happens and the result is exactly what you see when you run your code. Commented Dec 30, 2022 at 0:20
  • 1
    Ah - missed the overwriting part of your response. This does indeed make sense. Thanks! Commented Dec 30, 2022 at 1:00

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.