How to prevent grep --color from botching the output when a carriage return from Windows line ends is matched?

Question

I have a file with some lines likes

x
y

and when I run

grep -E "x$" filename.txt

it does not match anything. As vi shows ^M in the end of most lines, I guess the problem is mixed Dos and Unix newlines and grep autodetecting the format.

I tried

grep --color=never -E "x.$" filename.txt

to match the extra \r in the end of the line and it works, but it prints a single \r and thus breaks when, for example, --color=always adds terminal control characters.

What I need is an option, to match both \r\n$ and \n with $.

Hexdump of an example file:

00000000  78 78 78 78 78 78 78 78  78 78 78 78 78 78 78 78  |xxxxxxxxxxxxxxxx|  
00000010  78 78 78 78 78 78 78 78  78 78 78 0d 0a 79 0a     |xxxxxxxxxxx..y.|

You can see the DOS line end and the unix line end. The output of grep -E --color=always "x.$" seems to be empty when grep -E --color=never "x.$" prints the line, probably containing the carriage return \r that is matched by the ..

Answer 1

If you are using GNU grep, you can use the \s symbol from PCRE which matches any whitespace, so \s* will match 0 or more whitespace characters:

$ printf 'x\r\nxx\n' > file
$ grep --color=no -P 'x\s*$' file

xx

Note that what looks like an empty line isn't actually empty, it's the \r which is causing the terminal to move back and overwrite the x^*. You can see it in action with od:

$ grep -P 'x\s*$' file | od -c
0000000   x  \r  \n   x   x  \n
0000006

If you don't have GNU grep, you can use the POSIX character class [:space:] in the same way:

$ grep 'x[[:space:]]*$' file | od -c
0000000   x  \r  \n   x   x  \n
0000006

Removing the \r is also easy with standard tools like tr or sed:

$ tr -d '\r' < file | grep 'x$'
$ tr -d '\r' < file | grep 'x$'
x
xx
$ sed 's/\r//' file | grep 'x$'
x
xx

---

^* _{Note that, as explained by @dave_thompson-085, this only happens because I have my grep aliased to grep --color=auto which means that color codes are printed around the x, and that's what is causing the x to be overwritten as the \r causes the terminal to move back the cursor, so the x is then overwritten by the non-printing color escape codes.}

Answer 2

What I need is an option, to match both \r\n$ and \nwith$`.

You can do it with:

$ grep 'x^M\?$' infile

To type the '^M' you should hold the ctrl key, press V, then press ctrl again and press M on the command line. A ^M would appear, encoding a carriage return character.

Please understand that the output of grep will still contain a carriage return. And that that character tells the terminal to return the position for printing to the start of the line and keep printing from there. That might cause weird consequences.

What I would recommend is to clean the file with dos2unix.

dos2unix infile >clean.infile

Or even (just as an example, please don't cat the file, dos2unix is capable of reading it directly):

cat infile | dos2unix | grep 'x$'